Diary

Internal Reports

Completed Papers

Papers in Progress

Conference Presentations

Site Visit Reports

Main DLI Page

Web Client- DeLIver

Usability Test Results for the DLI Iodyne System, 12/ 96

Conducted by the Social Science Research Team

prepared by Cecelia Merkel

• Strength
• Weaknesses

• Search Window
• Results/Hits Window
• Metaview
• Title
• Author
• Subject
• Full Text
• Thesaurus
• C-space
• KWIC
• Icons

System Strengths
       

There were two issues that emerged when subjects indicated the strength of the system: (a) admiration for the search capabilities of the system, and (b) the layout of the search results window which allows a user to keep track of the searches they have performed. A common reaction to the system was that subjects admired the search capability of the system because that they could get at information in the testbed using multiple paths. The subjects could use the more advanced features of Iodyne such as the thesaurus or the different search tools to answer the same question in different ways. The subjects also liked the layout of the search results window because it allowed them to look at the different searches that they had performed all at one time. This allowed them to stay organized and keeping track of the terms and combination of terms that they had tried.

There were many other features of Iodyne that subjects liked. One of the features that they felt was the most useful was the ability to save searches. Others subjects mentioned that they liked being able to search multiple databases at one time, the drag and drop searching, being able to see a list of authors using concept space, searching the beginning of titles, and the short display which included the abstract.

System Weaknesses

Many subjects expressed frustration with the system because it was not intuitive, they were not sure how to use all of the system's features, and it was difficult to navigate the interface. Overall, many of the subjects commented that feeling lost using the system because they were not sure what steps they need to follow to perform a search. One subject said that if he/she was on his/her own he/she would not have know where to start and would have probably quit. Another subject said that he/she would need to develop a mental model for the system but that it was not easy. Subjects also commented on being unsure of how many of the system features actually worked especially the way that searching was being performed. There were problems in navigating the system in terms of knowing how to access the system's features, knowing what could be clicked on to get more information, and utilizing the drag and drop functionality. Overall, subjects expressed the desire for a system that felt integrated and that gave them hints about what steps to follow to complete a search.

Search Window

Problems Combining Terms

• drawing of trees to combine terms not obvious
• the process of drawing the branches was awkward resulting in branches that did not link the appropriate terms
• the redrawing process to remove a branch was not intuitive
• the redrawing process to remove a branch was awkward, subjects often drew new branches instead of removing them.
  


Manipulating Search Terms


• subjects wanted a way to remove all of the search terms at one time (they were looking for a button to do this)
• search terms were hidden underneath the listing of hits once multiple searches were performed
• there was difficulty editing terms on the search window
• the history feature of each search term was difficult to find

Performing Searches



• subjects were unclear about where they could perform Boolean searching (some subjects wanted to include multiple terms when entering terms rather than drawing trees to combine terms)
• subjects were not always clear on when an "and" search or an "or" search was being performed.
• subjects were not always entirely clear when a search was actually performed - they wanted something to happen to indicate that a search was completed
• subjects did not always complete a search because they did not select the radio button for the new term that they entered (so the system did not perform a new search, it was the old results)
  

Results/"Hits" Window

• issues included confusion about how to bring up information about one of the articles that the user was interested (the short view with abstract) in and how to interpret the results that the subjects received.
• it was not clear that you could click on the titles to get a short view of what the article was about
• there was s a lack of consistency in coloring of items that could be clicked on for more information in the metadata view as compared to the results/hits window
• the labeling of hits was confusing. Many subjects thought that they only had 100 hits when in reality that was the limit of the system.
• subjects had trouble recognizing the status window so they were not always sure which database they were connected to and the number of hits in each
  

Metaview of document

• the main problem subjects had was in finding the further information provided by this window.
• subjects did not immediately understand that they could click on the title and get the full text of the article.
• subjects expected the cursor to change, as if you were in a hypertext environment, to indicate that you could click on the title.
• difficult to find the option to list the figures, tables, and author affiliations Many of the subjects went to the full text of the article to find the figure because they did not realize that this was an option in this window.
• the abstract window should be labeled
  

Title Search Box

• subjects did not seem to have very many problems with this window but there was some confusion understanding the specific search options that were available.
• subjects were not sure which titles were being searched at first (the journal or the title of the articles) so this needs to be labeled.
• subjects were not sure how they should use the search beginning of the title option or the difference between match as whole word and match as whole phrase.
• subjects were not sure if they could enter multiple terms using Boolean searching.
• the match phrase option seemed to indicate to some subjects that you could enter multiple terms.
  

Author Search

• subjects did not know how to enter the author term (ex., last name, first name)
• subjects were unsure of how to use punctuation (they couldn't)
• subjects were not sure if capitalization mattered.
• The example that was used to assess this box was to pull up articles by H. Chen. Most subjects were not able to pull up only titles by H. Chen and they were not even really aware that they pulled up unrelated names until this was pointed out to them.
• subjects did not know what the option search stem word meant or the difference between stem word and full author name.
  
• one user expressed an interest in being able to restrict the results from this window such as by journal or year as in the full text search.
  

Subject Searches

• subjects were not clear which fields were being searched
• some subjects believed that the default was to search everything and they could not tell the difference between a full text search or a subject search.
• some had difficulty looking at the window knowing what to do because it wasn't clear that they should enter a search term.
• subjects, at first, were not clear on the difference between enter and search but once they tried it they seemed to understand.
• some wanted to perform Boolean searching entering multiple terms joined by an and/or/not
• subjects had problems telling when a search was performed from this window. They seemed to think that by hitting the search button a search would automatically be performed when they really had to select the term in another window. Some kept getting zero hits when using this window because they did not really perform a search.
  

Full Text Searches

• there was a lot of confusion between this type of searching and the subject search
• some had difficulty finding the option to limit their search to specific fields
• in general, subjects found this menu easy to use and they liked being able to click on only the fields they were interested in.
• subjects were not clear on the differences between match whole word and phrase.
• subjects wanted to be able to do Boolean searching and some tried it by linking terms with an "and" phrase.
  

Thesaurus

• subjects had difficulty finding this feature and sometimes had to be prompted to perform a subject search.
• without prompting people tried C-space and looked at the top menus to find the thesaurus.
• some subjects appreciated being able to see the hierarchy of terms because it gave them a better understanding of how terms were related. These subjects tended to be familiar with a thesaurus and its language such as related terms, broader terms, narrower terms.
• other subjects became confused when using the thesaurus in terms of understanding what they were seeing and in terms of manipulating the terms using the thesaurus.
• subjects found it difficult to understand what was happening when they clicked on a term and the display changed.
• subjects wanted a go back button so that they could see the display previous to the more narrow search.
• subjects did not always find the history button
• the history button repeated terms.
• subjects liked the chronology.
• most could understand the triangles.
  

C-Space

• most subjects did not understand what concept space was or how it worked.
• the labeling was confusing because they did not know what c-space meant.
• there was difficulty detecting the ordering of terms and some did not discover that there was a listing of authors
• subjects expressed a desire to know how the results were achieved - they did not trust it because they did not understand it. "It's just like a list of words".
  

KWIC/Keyword

• Subjects were not sure what this was or why it was there
• Subjects expressed frustration over the way the windows were displayed. They wanted the windows that were active to be on top and the inactive ones to fall to the bottom.
• subjects expected hits if a term was in the KWIC display which did not always happen

Icons

• subjects seemed to like the fact that they could keep a history of their searches using the index icon.
• they did not see much of a need for the garbage can because they wanted to be able to get rid of terms in the search window.
• many commented that they did not know what the icons meant and either asked us about them or ignored them.

Connection to Repository

• subjects need to be prompted to select a repository
• subjects need a clearer status bar reminding them which repository they are searching as they go along
  

Explanation of How System Works/Help Feature

• subjects admired the capability of the system but they were not clear on what all the features were, which fields they were searching, the form for entering terms, or the steps to take to complete a search. A help system might be useful to combat these problems.
• there needs to be an explanation of which fields are being searched for each menu item (author, subject, text, etc)
• an explanation must be provided about the differences between subject and text searching.
• an explanation must be provided for concept space and the thesaurus in terms of how it works, how it might be used, and which fields are being searched.
• the should be a menu item or a more obvious way to access the thesaurus
• Subjects did not associate the label C-space with concept space suggesting the need for more explanation or a different label.
• an explanation should be provided of terms used in the thesaurus. In particular less knowledgeable subjects did not know the meaning of some of the terms used (such as "related terms" or how this could impact their search.
• subjects were also not sure about specific features such as how to use the KWIC display so a help option would be helpful.
• subjects wanted a window that would give them hints as to what they should do next or what fields they were being searching.

Searching/Boolean

• the search trees were not obvious to most users so either an explanation or another way to combine search terms is needed
• many subjects wanted help in terms of knowing how to perform a search especially those familiar with Boolean searching.
• subjects were not sure where they could use Boolean searching. Those familiar with Boolean searching wanted to be able to type in and/or/not within each particular window (Ex. Author, subject) rather than drawing trees to connect terms.
• it was not clear to the subjects if they could enter multiple terms in each window. Within each window give an example of the preferred format for conducting a search (ex. Author Search, use chen h).
• there needs to be an explanation of terms such as match as phrase, etc.
• subjects also expressed a desire to be able to do two step searching by narrowing down the hits that they retrieved by date or by journal right in the window telling them the number of hits.
• there needs to be an indication that a search was performed - sometimes subjects did not select a term in the search window so they did not really perform a new search

Navigation

• subjects had a lot of trouble managing multiple windows and expressed desire for either a central management feature or the ability to cascade windows.
• subjects also wanted a way to close all to the windows quickly.
• the icons were not intuitive, many did not notice them or asked the researchers what they did.
• subjects also had problems finding the thesaurus, it could use its own button if this is a featured part of the system.
• In general subjects were not clear where they could drag and drop.
• overall, subjects wanted to see an integrated system, it was not clear how all the pieced fit together. One user suggested a big Iodyne window with all of the other windows fitting within the larger window.
• subjects wanted the to work like a hypertext environment in terms of what they could click on. They expected the cursor to change to a hand when it was something they could click on.
• subjects also expected windows features like having the cursor change between windows and change at places where you could change the size of the window.
• subjects also expected to be able to click on a window and have it come to the front of the display (this was a problem with the KWIC window).
  

Status Bar

• subjects wanted a status bar telling them what the system was doing. They wanted to know what repository was being searched, when a search was completed, and which fields were being searched.
• there needs to be a clearer indication of the number of hits that subjects obtained. Many said that they had 100 hits but in reality there were many more hits.
• some subjects expressed a desire on this status bar to have hints of what steps to take.
  

Consistency

• In the metadata window articles that were hypertext were in blue but in the results window they were black.
• Subjects associated using this system with hypertext so they expected a visited article to change color.
• The drag and drop features did not seem consistent because at times you dragged a term into the box below where you could type a term in and others you dragged it to where you typed the term.
• There was an inconsistency in the search fields, for example, no radio buttons were used in the author field but they were used in other areas.
  

This is a powerful system that lets users perform searches in multiple ways and gives them different ways of narrowing or broadening their search. Subjects recognized and appreciated what the system could do but had trouble knowing how to make use of all its features. One subject, a reference librarian, said that he/she would not recommend the tool for the average user. He/she would use it primarily when they needed to do a power search. Overall, subjects need help in understanding the features of the system, how they work, what is being searched, and in analyzing the information that they do retrieve.