Diary

Internal Reports

Completed Papers

Papers in Progress

Conference Presentations

Site Visit Reports

Main DLI Page

Web Client- DeLIver

Usability Tests for Custom Client, Testbed Team, DLI - Round 2. 12/ 96

Prepared by Emily Ignacio

December 16, 1996

I. Methods:

We invited four people to do a series of tasks. Two were librarians, two were engineers. Following Monk et al., we asked them to "think aloud" as they performed each task, and asked them to comment on what they were doing, what they were seeing and thinking about the interface as they proceeded. Two observers took session notes as the participant talked. This summary of the results are from our observations of three users. (One user was not able to do the tasks because the system had crashed.) At the end of the session, the participant answered a set of general questions about the interface and the usability test itself. These are included at the end of this report.

We hope that this summary of the results of this round of testing helps you develop future versions of the interface. If you would like any of the members of the social science team to provide you with informal input while you are between versions, please let us know. We will gladly help you out.

note: "**"s indicate problems that were noted on the first test of this interface in the spring of 1996

II. Boolean Searching:

Things they Did:

• did not find the AND/OR buttons on the bottom
• tried to type "AND" in the search window
• consistently thought that "add" meant "AND"
• typed two terms in the search window instead of using AND/OR buttons

May be due to the layout:

• not sure what was searched didn't notice "OR" next to the search windows
• not clear why there's a scroll window next to the ADD button when it only contains "OR"
• many did not notice the AND and OR buttons on the bottom of the form because they thought that all the tools for searching would be above the horizontal line and all the displayed stuff was at the bottom

III. Comments/Suggestions:

• "The form should have more emphasis on OR/AND commands; should move the ADD and OR buttons next to the search window."
• "The scroll window next to the search box should contain both AND and OR so we can choose which one we want."
• "Should have a NOT option so can eliminate similar terms, e.g., MIT "not" MITI."

IV. Problems with Information Given/Using the Interface Efficiently

• HINTS aren't very helpful **
• want case sensitivity **
• not sure what the occurring box (Word Wheel) is telling you; need more information about what it does
• for one of the tasks, they could not distinguish between the authors that were displayed (e.g., not sure which M. Smiths were brought up.) They could not find an easy way to distinguish between them.
• manually clearing terms is inconvenient
• "Can we search on initials? If so, how?"
• frustrated because came up with wrong "Summers" ; gave up
• not clear what is being searched; wants a status report that says what is being searched
• not sure what "listing of words box" actually does
• not sure if numbers next to the words indicate how many articles there are or instances the word appears
• "Why does it have to give you the total for each term?" subtotals don't make any sense
• not sure why people would want to go to the Table of Contents
• not sure if it's possible to look at full text along with Table of Contents

V. Comments/Suggestions:

• add a "CLEAR" button, so people don't have to manually clear items they wrote on the search form
• wants to be able to click on a name from the alphabetical list
• wants case sensitivity
• not sure why you need so many forms - just need one
• have hints next to each button, but be explicit when explaining what to do

VI. Problems with Layout/Color

• fonts too small **
• not sure where to enter search term at first
• person tried to click on red underlined words (on full text form) because they look like links, but nothing happened **
• need to make it clear that it's displaying 25 hits at a time; perhaps splitting results into two lines would help.
• e.g., 155 matches
• Displaying 1 - 25
• Windows pop up in different places each time
• not sure what the dot next to the AND term meant - thought he should clock on the button to "turn on" the AND function
• the "synonyms and related words" function doesn't do anything, though they look like links
• color-blind people would have trouble reading green and red against a blue background **
• both "exit system" button and "hints" are blue, but they are unrelated commands **
• not sure what any of the icons mean; only the checkmark makes sense **

VII. Comments/Suggestions:

• in author search form, want a button to clear entry
• want to be able to combine search term with author's name in the author form
• black against blue is easier to read **
• should make related commands have the same colors, and different commands (such as "exit system" and "hints" have different colors
• "Should use color consistently if it has meaning or don't use it at all." **

VIII. Problems with Language

• thought that "QUIT" was pressed on the Top Screen, it would take him to Netscape
• not clear that "change of view" applies to each element; thought it might apply to whole search
• not clear what "View Short Entry" means
• not sure what "# of hits" means
• not sure what using truncation would do
• not sure what "search full text" means at first

IX. Comments/Suggestions:

• add more information to "view short entry" window so person knows what it is.
• Initially, they thought it meant that it would automatically take them to the Panorama version, which they said, would be "a pain"

X. Problems with SGML/PDF Formats

• some preferred PDF format over SGML format because the pictures were clearer and the figures are embedded in the text
• easier to zoom in/out
• easier to page through
• others didn't know what you could do with the PDF version because it's not interactive and can't be manipulated, so SGML is better in this respect
• but, SGML is better because it's interactive and can be manipulated more; you can't cut and paste in PDF
• when retrieving figures, had trouble with the SGML version
• didn't know what globe vs. icon did; found out using trial and error **
• footnotes aren't labeled as footnotes, so clicked on "1" to get figure 1, was confused when it took him to the references
• didn't know how to get back into the text after they found the figure
• figures are unlabelled - "How could you tell the difference between them?"
• want to be able to search for specific terms easily instead of having to spend time scanning the article

XI. Comments/Suggestions for SGML version:

• perhaps a second layer of searching within the article itself
• might be nice to have a count of how many hits there are per page when you search for a term
• not obvious that F1 is Figure 1; could be footnote 1
• Panorama is not convenient when there is uncertainty about if one retrieved the "correct" article
• SGML is easy to read, yet it's weird that the figures aren't embedded within the text and are bunched up at the end; not like printed articles **
• flashing of screens before retrieving an article is annoying

XII. General Comments/Complaints

• not easy to get into Table of Contents
• want case sensitivity **
• wanted to enter both terms on the same line
• tried table of contents option, but wasn't sure where to go from there, so person quit
• too few journals

Problem with Windows Environment

• unless familiar with Windows environment, would not know that you could change options in scroll windows
• person who is not used to Windows does not know how to move/close windows

XIII. Best Thing:

• thought it was good that people could search full text
• quickness of system

XIV. Worst Thing/Needs Changing or Improvement:

• In general, "Make the interface more intuitive."
• takes too long to figure out how the interface works
• "People just want to be able to get in and get their data, not to have to learn the whole system."
• don't want to learn interface; just want to be able to get in and get data
• "The [full text form] is not intuitive."
• not sure what using truncation would do
• not clear what is being searched; wants a status report that says what is being searched
• not sure what "listing of words box" actually does
• not sure if numbers next to the words indicate how many articles there are or instances the word appears
• the "synonyms and related words" function doesn't do anything, though they look like links
• the ADD button doesn't make sense; the bigger buttons on the bottom seem more useful, but they're not noticeable because of their placement
• AND/OR buttons on bottom should be next to the search window
• the scrolling window makes it look as if you can have another option, but you can't
• color-blind people would have trouble reading green and red against a blue background
• black against blue is easier to read
• both "exit system" button and "hints" are blue
• should make them different colors because the difference between the two is not obvious
• - should use color consistently if it has meaning or don't use it at all
• not sure what "number of hits" means
• "Why does it have to give you the total for each term?" subtotals don't make any sense
• not sure what any of the icons mean; only the checkmark makes sense
• he display results screen
• "view short entry" should have more information next to it; not sure what it's function is
• likes "previous/next 50"
• likes SGML better than PDF because it's easier to read and is more interactive
• better/more help
• no rating output in SGML ; how many times did a word appear in the text?

XV. Comments/Suggestions:

• layout (see above)
• e.g., pop-up menu with AND/OR - should let people be able to type in AND/OR
• consistent color
• make buttons consistent
• first screen (3 buttons)
• not sure why people would want to go to the Table of Contents
• not sure if it's possible to look at full text along with Table of Contents
• not sure what "search full text" means at first
• have hints next to each button

XVI. Comments about the registration form:

• about affiliation?: "If I work downstairs, does that mean I'm affiliated?"
• address?: which? University or Home?
• about work?: which kind of work
• thought it was a bit long

Appendix A: The Instrument

Search Tasks (35 mins.)

1. Find all articles with "GaAs" or "InP" in either the title or abstract.

2. Find all articles written by M. Smith.

3. Find all articles written by someone at MIT (Massachusetts Institute of Technology).

4. Find all articles written by Nick Holonyak that include the term "Bragg reflectors" anywhere in the article.

5. Find all articles in which C. J. Summers appears as both an author and someone cited in the bibliography.

6. You met an interesting guy at a conference and you want to see what he's written. Try to find his articles. All you remember is that his last name is Chen and he works somewhere in Austin, Texas.

Do a search on a topic or author which which relevant to your current work. Find 3 articles you would want to print out. Examine the full texts of these articles.

Appendix B

Wrap-Up, Debriefing

Please offer any comments you have about the DLI testbed. We may ask you t comment on specific actions you took during this session. We'd also like your opinion on the following points:

1. What was the best thing about this prototype system?

2. What was the worst? What caused you the most problems?

3. What most needs changing or improvement?

4. Do you have any other suggestions for improving the system's functionality or interface?

5. What did you think of this usability test session?