Usability Study Index
Comments
“I think it was very helpful. I could- I would rely on this. I know I could rely on this if I was in this course specifically or another course. So. I think it was very helpful.”
“I don’t have any questions, but I do like this. I do want to know if this is going to be uploaded to where we can use it.”
“I do think it seems like a really good idea. I think it would be really helpful, especially for a lot of the starting out classes that people struggle with that everybody has to take. I do think that it’s a good idea, and I think that it’s something that I would personally use and I could see like recommending it to other people who were having issues with their classes.”
Article
Ray, T. (2024). Usability, visibility, and style: LibGuides usability testing to support students' needs. Southeastern Librarian, 72(4), 47-60. https://doi.org/10.62915/0038-3686.2108
About
In the Fall Semester of 2022, VSU's Virtual Service Librarian, Tera Ray, conducted a Usability Test on Odum Library's LibGuides to develop a set of best practices and find ways to enhance our guides for students. Research questions were developed at Reference Meetings with input from all Reference Librarians. This guide includes information about participants, methodology, and the study's findings. This study was IRB-exempt.
For any questions regarding this study, or the findings, please contact Tera Ray at taray@valdosta.edu or 229-259-7752
Methodology and Scenarios
Methodology
Participants completed three usability tasks while screen-capturing software recorded their screen and audio. Pre- and post-task interviews were conducted to gather demographic information and base knowledge of the library.
The researcher created two guides for the study based on current course guides. The guides are identical to one another outside of the guide layout. Guide one uses a tabbed or top navigation layout, while guide two uses a side navigation layout. The home page of both guides includes a welcome message, a Live Chat box, and a box with a link to our Using the Library FAQ Guide.
Other pages in the guides include:
-
Databases. A list of psychology-specific databases as well as a list of general databases
-
How to Find Articles. Explains how to use and select databases, use Discover, and explains basic search strategies.
-
How to Find Books. An explanation of GIL-Find@VSU and a live widget allowing the user to search GIL directly
-
Evaluating Journal Articles. A longer text box with valuable information for users on how to assess their resources
-
APA. A list of APA manuals in books, links to websites about APA citations, and DOI resources
Scenarios
Scenario 1
You are in a Psychology class and have to write a literature review. Your topic is how screen time affects the mental health of children. Find a scholarly article on this topic.
Research Question: How do students use guides?
It is believed that students use Research Guides for links to subject-specific databases. This scenario tests how participants will use the guide to look for resources.
Scenario 2
You've found an article that you like, but you're worried it may be biased. How would you use this page to help you find out?
Research Question: What is the tl;dr (too long; didn't read) threshold?
To complete this task, participants were expected to navigate to the “Evaluating Journal Articles” page and find the “Bias” section from the text.
Scenario 3
You are having trouble finding resources that relate to your topic. Where would you go for help?
Research Question: How do students use guides?
Several ways to get help from a librarian, including LiveChat, are listed on the guide. This task is designed to understand how students get help, where they go, and why.
Participants
A total of 10 participants, all enrolled in courses at VSU, were a part of the usability test. Five participants completed tasks on a guide with top navigation, while the other five performed the same tasks using a similar guide with side navigation. These were randomly assigned and the distribution of the assigned guides by student classification as well as major type is shown below.
Demographics
Participants testing the Top Navigation guide consisted of 1 freshman, 1 sophomore, 1 junior, and 2 seniors. Their major types were 1 in art, 1 STEM, 1 health/health sciences, and 2 in humanities and communication.
Those testing the Side Navigation guide included 2 freshmen, 1 sophomore, 2 juniors, and no seniors. Their major types were 3 in education, 1 STEM, and 1 health/health science.
Library Use
- 9 out of 10 participants reported visiting the library in person at least once a week
- 6 out of 10 participants reported using the library website on a weekly basis
Library Instruction
- 5 out of 10 of the participants reported having a Library Instruction at some point at Valdosta State University
- 1 freshman, 1 sophomore, 1 junior, and 2 seniors
Findings
Research Question(s)
Do students know about LibGuides/Research Guides? If so, how?
Findings
- None of the participants of the usability study reported ever using a research guide
- 3 out of 10 of the participants stated that they had heard of a research guide in the past
- From where?
- 1 from a previous institution
- 1 from a Library Instruction
- 1 from another student
- From where?
Literature...
- Most studies show that the biggest issue with LibGuides is that students don't know about them. Guides that have the highest usage are often connected to a course or an assignment and promoted by an instructor.
Dalton, M., & Pan, R. (2014). Snakes or ladders? Evaluating a LibGuides pilot at UCD Library. The Journal of Academic Librarianship, 40(5), 515-520. https://doi.org/10.1016/j.acalib.2014.05.006
Emery, J. L., & Fancher, S. E. (2016). Pay attention to the data behind the curtain: Leveraging LibGuides analytics for maximum impact. In R. L. Sittler & A. W. Dobbs (Eds.), Innovative LibGuide applications: Real world examples (pp. 113-127). Rowman & Littlefield Publishers.
Murphy, S. A., & Black, E. L. (2013). Embedding guides where students learn: Do design choices and librarian behavior make a difference? The Journal of Academic Librarianship, 39(6), 528-534. https://doi.org/10.1016/j.acalib.2013.06.007
Quintel, F. (2016). LibGuides and usability: What our users want. Computers in Libraries, 36(1), 4-8.
Research Question(s)
Which navigation style is preferred - top navigation (tabbed) or side navigation?
What is the tl;dr (too long; didn’t read) threshold?
Findings
Navigation Preference
- 6 out of 10 students preferred top navigation
- Wider view and columns
- ”I do like that it’s more fit to the screen… so having a wider view it’s more like a website.”
- Navigation location
- “[The navigation menu is like] you’re opening a file. And you know, one of of those Manila folders where there’s a tab at the top?”
- Aesthetics
- Participants noted that the side navigation felt “one-sided". Top navigation was appreciated for its symmetry.
- Wider view and columns
- 4 out of 10 students preferred side navigation
- Navigation location
- “…everything is in a stack … it’s just easier for me to read.”
- One participant noted that top navigation made the top of the screen seem crowded.
- Aesthetics
- ”It looks cleaner”
- ”I think [the side navigation guide] has a better look to it.”
- ”It’s bolder, so, I like the darker color. Kind of gives an eye-catching appeal.”
- Navigation location
Ease of Use and Success
Success Rate
Based on pre-determined criteria, each participant was recorded as either completing or not completing the task assigned.
Scenario 1: All participants in both guides completed the task.
Scenario 2: For top navigation, 4 participants completed the task. For side navigation, only 2 were able to complete the task.
Time Taken to Complete the Tasks
Each task was timed from the moment the researcher finished reading the task aloud to when the participant completed the task. Unsuccessful attempts were not included in the average calculations below.
Top navigation had a 15-second lead in completion time for Scenario 1, while in Scenario 2, side navigation had a 17-second lead in the time taken to complete the task.
Number of Clicks
The number of clicks a participant took to complete the task was calculated and averages were rounded to the nearest whole number. The researcher counted all clicks from the moment the user began the task, to when the task was completed. Unsuccessful attempts were not included in the calculation.
Both groups completed Scenario 1 with the same number of average clicks. For Scenario 2, participants using the Top Navigation layout completed the task in an average of 2 clicks, much less than the 5 click-average that Side Navigation participants finished with.
TL;DR Threshold — Task Two
Students were asked to find a section in the guide that would help them identify bias in an article.
- Most participants skimmed the material on the page until they found what they needed
- On average, participants who found the section on bias spent 30.6 seconds* on the tab
- *This excludes one participant who stopped in the middle of the tasks to ask questions about the guide that was not related to the task at hand.
- One participant navigated to the correct tab/page but did not find the section on bias. This participant stayed on the page for 19 seconds before clicking on the only link, “What is Peer Review,” that appears near the top of the page.
When asked what they thought about the amount of information, most participants stated that they thought it was the right amount. Only one stated that there was too much:
- ”There’s a lot of information. So maybe just certain links... something that could help that’s more official… I guess I didn't look over enough to understand it but, um, what is a peer review? Like, that’s a lot of information that’s given.”
Results of this task show that students will skim material to find what they need, thus highlighting the importance of using headings.
Literature
- There have been similar usability studies that have shown students being nearly split on the preferred navigation style (Barker & Hofman 2021; Chan et al., 2019).
- Other studies have shown a preference for one style over the other, with no clear consensus in the literature of which style is more preferred by students (Conerton & Goldenstein, 2017; Oullette, 2011).
- A consistent layout across guides is more important than the layout and navigation style of the guide (Cobus-Ku et al., 2013; Hungerford et al., 2010; Pickens, 2016; Stonseby & DeJonghe, 2013).
- Literature suggests that students prefer a simple, clean layout (Barker and Hofman, 2021; Conerton & Goldenstein, 2017).
- Several studies found that students did not care if they had to scroll or if there was a lot of information on the page, as long as it was well organized (Cobus-Ku et al., 2013; Pickens, 2016).
Barker, A.E.G., & Hoffman, A. T. (2021). Student-centered design: Creating LibGuides students can actually use. College & Research Libraries, 82(1), 75-91, https://doi.org/10.5860/crl.82.1.75
Chan, C., Gu. J., & Lei, C. (2019). Redesigning subject guides with usability testing: A case study. Journal of Web Librarianship, 13(3), 260-279. https://doi.org/10.1080/19322909.2019.1638337
Cobus-Kuo, L., Gilmour, R., & Dickson, P. (2013). Bringing in the experts: Library research guide usability testing in a computer science class. Evidence Based Library and Information Practice, 8(4), 43-59. https://doi.org/10.18438/B8GP5W
Conerton, K., & Goldenstein, C. (2017). Making LibGuides work: Student interviews and usability tests. Internet Reference Service Quartly, 22(1), 43-54. https://doi.org/10.1080/10875301.2017.1290002
Conrad, S., & Stevens, C.S. (2019). "Am I on the library website?": A LibGuides usability study. Information Technologies and Libraries, 38(3), 49-81. https://doi.org/10.6017/ital.v38i3.10977
Hungerford, R., Ray, L., Tawatao, C., & Ward, J. (2010). LibGuides usability testing: Customizing a product to work for your users. Retrieved from http://hdl.handle.net/1773/17101
Ouellette, D. (2011). Subject guides in academic libraries: A user-centered study of uses and perceptions. Canadian Journal of Information and Library Science, 25(4), 436-451. https://doi.org/10.1353/ils.2011.0024
Sonsteby, A., & DeJonghe, J. (2013). Usability testing, user-centered design, and LibGuides subject guides: A case study, Journal of Web Librarianship, 7(1), 83-94. https://doi.org/10.1080/19322909.2013.747366
Pickens, K. E. (2016). Applying Cognitive Load Theory principles to library instructional guidance. Journal of Library & Information Services in Distance Learning, 11(1-2), 50-58. https://doi.org/10.1080/1533290X.2016.1226576
Research Question(s)
How are LibGuides used?
What do students expect from a LibGuide?
Findings
Expectations
- Half of the participants believed Research Guides would contain information about how to do research, while the other half believed the guides would contain information on the topic with potential sources and links to other websites about the subject.
Use
- How would participants use LibGuides to find a scholarly article?
- Most participants (n=6) clicked on the “How to Find Articles” tab and then clicked the link to the library’s homepage, which happened to be the first link on the page. They used the Discover search bar to find an article.
- Only 2 participants used the Databases tab.
- 2 participants tried using the LibGuide search bar at the top of the guide to search for the topic.
Literature
- A 2021 card-sorting study conducted by Barket and Hoffman at Kennesaw State University found that participants expected to see both instructional content and resources in a research guide.
Barker, A.E.G., & Hoffman, A. T. (2021). Student-centered design: Creating LibGuides students can actually use. College & Research Libraries, 82(1), 75-91, https://doi.org/10.5860/crl.82.1.75
Several other unexpected themes emerged during testing that were not related to the research questions.
The LibGuides Search Bar
- 2 out of the 10 participants tried searching using the LibGuides search bar when asked to find an article.
- When asked to find a section for bias, one participant tried using the search bar to find an area in the guide that mentioned bias
- Note: This would have worked if the guide had been published publically. A guide that is published as private will not search within the guide.
- When asked to find a section for bias, one participant tried using the search bar to find an area in the guide that mentioned bias
- When asked what they would change, one participant replied that they would fix the search bar to accurately search the guide.
Changes Made
Previously, when searching in a guide or on LibGuides in general, the top of the guide would only show the number of results. To help patrons understand the function of the search bar, the wording was edited so that it still shows the number of results, but over that information, the guide reads:
If you want to find articles or books, try searching in Discover on the Library Homepage
This should give patrons more information on where to go for what they want to search. The phrase appears both on the guide search page as well as on the system search for all VSU's LibGuides.
Images, Aesthetic, and Interactive Features
- When asked how they would improve the guide, participants stated that they would add images with one mentioning a need for more color.
- ”Maybe add like, pictures or something? Or like, some type of imagery that would just draw people into it more…”
- ”… something that attracts me, that I want to see… like a slideshow piece on the side, like giving a brief of the library.
- ”…maybe like, add a picture or make it pop more rather than just like, black and white with a little bit of red.”
- ”…[add] a picture related to the course or just like graphics … [to] invoke more connection between the user and the computer.”
- Screenshots on the How to Find Articles page were appreciated by many.
- The aspect of the guide that received the most positive feedback was the interactive widget that allowed patrons to search the catalog without having to leave the guide.
Non-Library Resources
- Participants were asked where they typically go to find resources for their research and research papers. Responses are listed below.
- 36.4% Library resources including the library website, GALILEO, and databases provided by the library.
- 36.4% Google
- 18.2% Britannica
- 9.1% Google Scholar
Getting Help
- Most participants stated that they would try to figure out an issue themselves before asking a librarian for help.
- When asked where students would go for help with finding resources, contacting a librarian for help was mentioned a total of 12 times.
- Chat - 4
- In person at the Reference Desk - 4
- Phone - 3
- Research Appointment - 1
- Participants also stated they would ask their instructor (n=2) or their classmates (n=1).
