Abstract

This report discusses the findings of three usability tests that were conducted to determine how workability and user-friendliness of a new website. This website was specifically constructed for the editorial staff at The World & I Online, an online educational magazine based in Washington, D.C. Its purpose is to serve as an online archive of electronic links to websites that the magazine staff uses.

World & I staffers perform research on the Internet to find new content and new ideas for the magazine. There are a number of Web resources that the staff members use on a regular basis either to obtain new material directly or to generate ideas for new topics to cover in future issues. Some of these online resources are archives or databases where staffers can search for articles on a particular topic, some are academic resources with interesting text or images, and some are websites that discuss helpful instruction methods for teachers.

I decided that a Web-based resource archiving many of the most important Internet resources used by World & I staffers could be very useful to the editorial team. If the website listed electronic links to all of these online resources and categorized them by what type of resource they were, staffers could find these academic sites much faster. By organizing the links in a comprehensible way, staffers could perform research more efficiently and generate more ideas for new content in the magazine.

The findings of the study indicate that some of the main difficulties that users encountered were that they were not always certain about what content the Web resources contained, sometimes needed help navigating back to the home page or to lists of academic think tanks, and sometimes had trouble reading down the lists of hyperlinks because they were close together. The website’s overall appearance and design were well-received, and the links to outside websites for research were generally easy to locate and access.

Major recommendations for changing the site include placing some links under more than one category of resource, creating an additional page to list only links to think tank websites, including brief summaries of all resources beneath their hyperlinks, making more space breaks between hyperlinks on the lists, including a link back to the home page on each internal page of the site, and providing an e-mail address for anyone who wants to ask questions about the site.

I. Introduction

A. Purpose

This report is intended to convey the results of a series of tests designed to determine the usability of a Web-based archive of research sources for a magazine editorial team. The following information describes the findings from the tests and the conclusions drawn about what kinds of changes need to be made on the website to make it more usable and helpful for its target users. The intended audience for this report would be the publisher of the magazine so that he can see how changes made to the website would make it more useful to the editorial staff. I am also part of the audience since my purpose is to outline the changes that I want to make to the site based on my findings in the usability tests.

B. Definition, Description and Background

The usability tests covered in this report were conducted to examine the ease with which editorial staffers could use a Web-based archive of electronic links to academic resources on the Internet in their research. This website was specifically constructed for the editorial staff at The World & I Online, an online educational magazine based in Washington, D.C. The World & I Online is a monthly publication aimed at teachers, students and scholars that features a wide variety of academic, scholarly and journalistic content that can enhance people's educational experience.

World & I staffers continually perform Internet research to find new content and new ideas for the magazine. There are a number of Web resources that the staff members use on a regular basis either to obtain new material directly or to generate ideas for new topics to cover in future issues. Furthermore, there are many more academic and journalistic websites that are useful to read and explore when it comes to content development. Some of these online resources are archives or databases where staffers can search for articles on a particular topic, some are academic resources with interesting text or images, and some are websites that discuss helpful instruction methods for teachers.

I decided that a Web-based resource archiving many of the most important Internet resources used by World & I staffers could be very useful to the editorial team. If the website listed electronic links to all of these online resources and categorized them by what type of resource they were, staffers could find these academic sites much faster. By organizing the links in a comprehensible way, staffers could perform research more efficiently and generate more ideas for new content in the magazine. They would not have to try to constantly remember what resources they need or what their Web addresses are because all of the resources would be stored in one place. I wanted this site to be successful by making research easier for editorial staffers at The World & I Online, a specific audience and a specific enterprise (Lynch, Horton 14).

With this specific purpose and audience in mind, I designed a website where I could list links to the key resources that the magazine staffers use. I created three main categories of research sources and divided them into three separate pages of lists of Web links that can be accessed from the site's home page. The categories include: · Search Databases - sites where editorial staffers can find content through search functions and find information on topics by typing in keywords. · Academic Sites - resources containing academic, scholarly and journalistic content that can either be reprinted in the magazine or can serve to inspire ideas for new content. · Teachers' Tools - sites that feature lesson plans, teachers' guides, teaching tips, and other tools that can help teachers to present their course material in the classroom.

After designating these categories, I took some samples of Web resources commonly used at The World & I Online and placed their Web links under the categories where they fit best. I created three separate pages to list the Web links according to their type of resource.

C. Method of Inquiry

To test the usability of the website, I asked three users to participate in the usability test and to perform designated tasks associated with a scenario involving editorial research. As the moderator of the test, I described the scenario to the user and asked the person to perform a given research task to address the scenario. I used a moderator guide that I had previously prepared to describe the scenarios and tasks. When I originally composed the moderator guide, I designed the scenarios and tasks based on my personal experience as an editorial assistant at The World & I Online and on my observation of what other staffers did in their online research. This way, the tests would accurately reflect the work done at The World & I Online and would be consistent with the experience of the target audience. My first user was a classmate at school and my second and third users were editorial staffers at the magazine.

D. Limitations and Scope

The study is limited by the small number of participants in the testing--three participants in all. A wider range of participating users would probably yield more findings about the website's usability, but not all potential users could take part in this single study at the same time. In drawing conclusions, the study is confined to examining the findings of the usability tests involving only three individuals.

The study is also limited because the scenarios and tasks outlined in the moderator guide are only a few of the many possible tasks that a staffer may perform during research. This study could not anticipate every conceivable scenario that an editorial assistant might encounter in the course of doing research or the associated tasks. Therefore, for the purposes of this study, the tests were confined to three major scenarios with two tasks assigned to each one.

E. Recommendations

The results of the tests indicated that the users had a generally positive experience with the website as a whole. Their feedback suggests that the basic format and design concept should remain the same. · The background color and the overall appearance of the site works well for the users, and the three main categories of resource links is very helpful in leading them to the Web resources that they want to visit. · The links are functioning properly, and users seemed to like the way in which the links took them to Web resources (by opening a new browser window alongside the window displaying the archive site). · The link to the home page of The World & I Online is also useful to staffers and will remain.

Some parts of the website should be changed based on the test feedback. · Some users believed that some of the Web resource links should be displayed under more than one of the three major categories. Some resource sites blend traits of academic sites and search databases, so some of the links to sites like these should appear on both the page featuring search databases and the page featuring academic resources. · There should be an additional page linking from the academic sites page that displays only Web links to academic think tanks. Since there are many types of think tanks that contain interesting scholarly material, it would be helpful if staffers could see them listed all together on one page. The think tank list will be a sub-category branching out from the academic sites page. · The users liked the idea of including one-to-two sentence descriptions of the academic resources beneath each Web link, so it is important to include those summaries with each hyperlink. · The home page should include a link to my e-mail address so that users can contact me with questions or comments about the site. · Each Web page that is part of the website should have a hyperlink at the bottom leading back to the home page. · Make more space between listed hyperlinks on the Web pages to allow for easier reading by users.

By incorporating these changes into the website, I can build on the traditional concept of a print archive for researchers and scholars by making an electronic archive that can be used by the editorial team of an online publication (Bolter 37). Combining modern science and technology with time-tested principles of good publishing will produce a product that is useful to its target audience (Bolter 19).

II. Collected Data

A. First Topic for Investigation - User #1

1. First Sub-Topic - Searching for Content a. In the first scenario, User #1 wants to use a search database to find articles to reprint in a future edition of The World & I Online. b. User #1 was asked to perform two tasks: 1) Locate a Web link on The World & I Online's Resource Links Page that leads to a search database on the Web to find articles for the magazine. 2) Describe how easy or hard it is to perform task #1 using the website. c. User #1 found the United Press International archive site by looking under the Resource Links Page category Search Databases. The user was then able to use the UPI archive's search function to call up many articles that matched with keywords typed in. User #1 found the search databases easy to find on the website and easy to use to find article matches. She suggested including summaries describing each search database underneath the hyperlinks, and also suggested mentioning the fact that World & I staffers need a username and password to access the UPI archive. A link leading back to the home page would also be helpful for all internal pages. d. Judging from User #1's feedback, it is important to keep the user as informed as possible about the resources listed on the website. Therefore, each Web link will include a short but helpful description of the resource that it represents. A link to the home page will also be added to help the user with navigation.

2. Second Sub-Topic - Academic Resources a. In the second scenario, User #1 wants to visit an academic resource on the Web to get ideas for new topics to feature in The World & I Online. b. User #1 performed two tasks: 1) Locate a Web link to an academic website that you want to read for story ideas. 2) Describe how easy or hard it is to find academic sites using the Resource Links Page. c. User #1 was able to easily access the academic and journalistic sites listed on the Academic Sites page. These sites included publications such as The Washington Times, The Washington Post, and The New Republic. User #1 had no trouble exploring them for content. She suggested that the Academic Sites page include some links to more academic institutes, research institutions, and other organizations involved in education and research. d. The website will include more links to prominent academic and scholarly organizations that would be useful to staffers.

3. Third Sub-Topic - Teaching Materials a. In the third scenario, User #1 wants to find ideas for how to improve the Teacher's Corner page at The World & I Online, where the magazine features teaching materials. To do this, User #1 needs to observe examples of teachers' guides and lesson plans on the Web. b. User #1 performed two tasks: 1) Locate a Web link to a site on teachers' tools using the Resource Links Page. 2) Describe how easy or hard it is to find teachers' tools by using the Resource Links Page. c. User #1 had no trouble going to the Teachers' Tools page to perform the task. She especially liked the Web link to The Educator's Reference Desk site because of the many lesson plans on multiple subjects. User #1 suggested that the Teachers' Tools page discuss what school subjects the Web resources cover, the grade levels of the lesson plans, and the types of teachers that the lesson plans can serve. She also suggested including my e-mail address on the home page in case users have questions about the site. d. Information on resources designed to assist teachers can be very useful to staffers trying to find ideas on how to communicate to teachers with The World & I Online. I will try to make the Teachers' Tools page indicate the relevant educational traits of the teachers' materials on the Web.

B. Second Topic for Investigation - User #2

1. First Sub-Topic - Searching for Content a. User #2 was presented with the same scenario as User #1. b. User #2 was presented with the same tasks as User #1. c. User #2 successfully accessed the Web resources on the Search Databases page and was able to find many articles by plugging keywords such as "Africa" into the search functions. She suggested that the Search Databases page include a link to The Washington Times home page because it has a search tool that the staffers use frequently. Other possible search functions might include Google and Wikipedia. User #2 also thought that short blurbs describing the resources were helpful. d. I originally put the link to The Washington Times under Academic Sites, so I will now also include it under Search Databases. It is also clear that the blurbs are treated as a very good idea in the tests so far.

2. Second Sub-Topic – Academic Resources a. User #2 was presented with the same scenario as User #1. b. User #2 was presented with the same tasks as User #1. c. User #2 had no trouble opening the Web links on the Academic Sites page and exploring the sites that I listed on the page. She suggested that the Academic Sites page include more scholarly and educational organizations such as the Smithsonian, and also that it would be helpful if there was a separate list of some organizations such as scholarly think tanks on a separate internal page. User #2 also thought that the YaleGlobal site and the Integrated Regional Information Networks site should both be listed on the Academics Sites page as well as on the Search Databases page. d. Additional academic resources will be added to the list besides the scholarly and journalistic publications that I already listed. Also, it seems that a new sub-category of academic organizations would make navigation easier for users. Finally, since YaleGlobal and IRIN can be classified as academic resources and search databases, they will appear under both categories.

3. Third Sub-Topic – Teaching Materials a. User #2 was presented with the same scenario as User #1. b. User #2 was presented with the same tasks as User #1. c. User #2 liked the lesson plans that the Teachers’ Tools page linked to, and she suggested that more lesson plans and teachers’ resources be included. For the site overall, User #2 suggested making more space between Web links to make them easier to read, and also providing a link to the home page on all of the internal pages. d. It is clear that the users would find it easier to use a link to the home page than to hit the back button. More breaks of space between the links and their blurbs along the lists would also make them easier to read.

C. Third Topic for Investigation – User #3

1. First Sub-Topic – Searching for Content a. User #3 was presented with the same scenario as Users #1 and 2. b. User #3 was presented with the same tasks as Users #1 and 2. c. User #3 successfully navigated the Search Databases page and was able to access all of the listed sites. She suggested that more space be made between the links and their blurbs so that users can better read the text and better find the resources that they want. Also suggested was that some of the bullets might be made bigger and some of the text in the links and blurbs be made smaller. d. The recommendation that there be more space between links and text came up in this test as it did in the second test. This indicates that users definitely can read the lists on the Web pages more easily with a little more space.

2. Second Sub-Topic – Academic Resources a. User #3 was presented with the same scenario as Users #1 and 2. b. User #3 was presented with the same tasks as Users #1 and 2. c. User #3 successfully accessed the academic resources on the Academic Sites page and had no trouble exploring those Web resources. She conveyed that a separate page for think tanks would be very useful to World & I staffers when they look up research sources. User #3 also recommended mentioning in the summary descriptions of some Web resources where staffers had blanket copyright permission to reprint articles. d. The feedback definitely indicates that a separate Web page for academic organizations like think tanks would help World & I editorial assistants. More resource links and summaries can be added to the list.

3. Third Sub-Topic – Teaching Materials a. User #3 was presented with the same scenario as Users #1 and 2. b. User #3 was presented with the same tasks as Users #1 and 2. c. User #3 successfully accessed the teachers’ Web resources listed on the Teachers’ Tools page and encouraged more electronic links to online educational sites for teachers. She liked the lesson plans and teachers’ guides included in the resources I named. d. Lesson plans and teachers’ guides are great resources for World & I staffers trying to develop content that will appeal to teachers, who make up a large share of the magazine’s audience.

Conclusion

The findings from the three usability tests indicate that the Resource Links Page primarily needs sufficient clarity about the listed Web resources—what content they feature and what their purposes are. This way, users will know where to find the types of websites they need for their research. By writing brief one-to-two sentence summaries and blurbs to go with each Web link, World & I staffers can carefully choose which Web resources would help them most when they conduct their research for magazine content.

Also, it is important to remember that not all Web resources have to appear under only one category; they may qualify to appear under more than one category of resource type. Navigation was not problem for users except that it would be helpful to have a link to the site’s home page at the bottom of each internal page. Likewise, the users found the overall appearance and design of the website appealing, but also recommended that there be more space between links and text so that users could read the site more easily.

Additional features such as a separate page listing think tank links and a link to my e-mail address were also commonly suggested in the tests. By incorporating these changes, the Resource Links Page would serve a valuable function of helping editorial staffers at The World & I Online to research and develop content for the magazine more efficiently and productively.

References

Bolter, Jay David. Writing Space: Computers, Hypertext, and the Remediation of Print. Mahwah, NJ: Lawrence Erlbaum Associates, Inc., 2001, p.19

Bolter, Jay David. Writing Space: Computers, Hypertext, and the Remediation of Print. Mahwah, NJ: Lawrence Erlbaum Associates, Inc., 2001, p.37.

Lynch, Patrick J. and Sarah Horton. Web Style Guide. New Hover: Yales University Press, 2001, p. 14.

Appendices

Appendix A. User Matrix

1. User #1 was a classmate who suggested that blurbs or summaries should be added to describe each Web resource, that more academic institutions should be listed as resources, and that teachers’ tools links should indicate what types of lesson plans were featured and what kinds of teachers they served.

2. User #2 was a World & I Online editorial assistant who recommended short summaries of Web resources, a separate page listing academic think tanks, more spacing between hyperlinks, links to the home page from other internal pages, and an e-mail address for users to contact.

3. User #3 was a World & I Online editor who suggested that I include short summaries of all Web resources beneath the hyperlinks, create a separate page for think tanks branching out from the academic sites page, indicate blanket copyright permission to reprint articles from certain sites, and make more space between hyperlinks.

Appendix B. Moderator Guide

Below is the moderator guide that I used in the usability tests to guide the users. Moderator Guide User’s Name: _______________________________ Date: _______________________________ Time: _______________________________ Hello. Thank you for participating in today’s usability test. You’re doing us a tremendous favor and we very much appreciate it. We’re developing a new website and we’d like your input on how easy it is to use the site. Your feedback is crucial to helping us know how we can improve our site. We want to make the site valuable to you and as easy to use as possible. Rest assured that your name will not be used in the final report on this usability test. Just to cover a few things quickly: • Have you ever participated in research such as focus groups, or anything like that, before? • How often do you use the Internet? What do you use it for? How long have you been using it? This is very simple – we’re going to pull up a Web page on this computer screen in front of you. I’m going to ask you to complete a few simple tasks, and comment on them as you go. Read out loud what you’re looking for, when you find it, and what you would expect to happen when you click there. The more you talk, the more information we get, and the more helpful you become. Please feel free to stop me and ask questions any time. Scenario #1 You want to use a search database to find articles to reprint in a future edition of The World & I Online. User Task 1A Locate a Web link on The World & I Online’s Resource Links Page that leads to a search database on the Web that you can use to find articles. Goal: Determine how easily users can find a search database using the website’s archive of Web links to online research tools. User Task 1B Describe how easy it is to read the list of search database links on the page. Goal: Determine whether users can read this archive list of sites and find the link they want by scanning the links (which are listed in alphabetical order). Scenario #2 Now you want to visit an academic resource on the Web to get ideas for new topics to feature in The World & I Online. User Task 2A Locate a Web link to an academic website that you want to read for story ideas. Goal: Determine how easily users can find an academic resource on the Web using the website’s archive of Web links to academic sites. User Task 2B Describe how easy it is to read the list of academic links on the page. Goal: Determine whether users can read this archive list of sites and find the link they want by scanning the links (which are listed in alphabetical order). Scenario #3 You want to find ideas for how to improve the Teacher’s Corner by observing examples of teachers’ guides and lesson plans on the Web. User Task 3A Locate a Web link to a website on teachers’ tools that you want to read for story ideas. Goal: Determine how easily users can find a resource on the Web discussing teaching tools by looking them up in the online archive. User Task 3B Describe how easy it is to read the list of teachers’ tools links on the page. Goal: Determine whether users can read this archive list of sites and find the link they want by scanning the links (which are listed in alphabetical order). We’re finished! Thank you so much for participating. You’ve been an enormous help, and we really appreciate it.

Appendix C. User Summaries

1. User #1 User #1 found the United Press International archive site by looking under the Resource Links Page category Search Databases. The user was then able to use the UPI archive's search function to call up many articles that matched with keywords typed in. User #1 found the search databases easy to find on the website and easy to use to find article matches. She suggested including summaries describing each search database underneath the hyperlinks, and also suggested mentioning the fact that World & I staffers need a username and password to access the UPI archive. A link leading back to the home page would also be helpful for all internal pages.

User #1 was able to easily access the academic and journalistic sites listed on the Academic Sites page. These sites included publications such as The Washington Times, The Washington Post, and The New Republic. User #1 had no trouble exploring them for content. She suggested that the Academic Sites page include some links to more academic institutes, research institutions, and other organizations involved in education and research.

User #1 had no trouble going to the Teachers' Tools page to perform the task. She especially liked the Web link to The Educator's Reference Desk site because of the many lesson plans on multiple subjects. User #1 suggested that the Teachers' Tools page discuss what school subjects the Web resources cover, the grade levels of the lesson plans, and the types of teachers that the lesson plans can serve. She also suggested including my e-mail address on the home page in case users have questions about the site.

2. User #2 User #2 successfully accessed the Web resources on the Search Databases page and was able to find many articles by plugging keywords such as "Africa" into the search functions. She suggested that the Search Databases page include a link to The Washington Times home page because it has a search tool that the staffers use frequently. Other possible search functions might include Google and Wikipedia. User #2 also thought that short blurbs describing the resources were helpful.

User #2 had no trouble opening the Web links on the Academic Sites page and exploring the sites that I listed on the page. She suggested that the Academic Sites page include more scholarly and educational organizations such as the Smithsonian, and also that it would be helpful if there was a separate list of some organizations such as scholarly think tanks on a separate internal page. User #2 also thought that the YaleGlobal site and the Integrated Regional Information Networks site should both be listed on the Academics Sites page as well as on the Search Databases page.

User #2 liked the lesson plans that the Teachers’ Tools page linked to, and she suggested that more lesson plans and teachers’ resources be included. For the site overall, User #2 suggested making more space between Web links to make them easier to read, and also providing a link to the home page on all of the internal pages.

3. User #3 User #3 successfully navigated the Search Databases page and was able to access all of the listed sites. She suggested that more space be made between the links and their blurbs so that users can better read the text and better find the resources that they want. Also suggested was that some of the bullets might be made bigger and some of the text in the links and blurbs be made smaller.

User #3 successfully accessed the academic resources on the Academic Sites page and had no trouble exploring those Web resources. She conveyed that a separate page for think tanks would be very useful to World & I staffers when they look up research sources. User #3 also recommended mentioning in the summary descriptions of some Web resources where staffers had blanket copyright permission to reprint articles.

User #3 successfully accessed the teachers’ Web resources listed on the Teachers’ Tools page and encouraged more electronic links to online educational sites for teachers. She liked the lesson plans and teachers’ guides included in the resources I named.