System Usability Evaluation

Goals

Strategy

Objectives

Levels of Evaluation

Measurement Approaches

Formality and Informality of Methods and Rationale

Societal and Cultural Factors

Desired Results (Information Sought from Evaluation)

Goals

• Determine if T/TAC staff can easily upload their training content to the T/TAC Online site and uncover any potential difficulties with the upload process.
• Determine the best practices for the inclusion of multi-media by establishing which content formats will best meet the needs of the staff and the site's audience.
• Assess the effectiveness of a pre-production workflow process designed to assist the T/TAC staff in the preparation of their content for uploading.

Strategy - Phase One

• Schedule six T/TAC staff at the GMU T/TAC office to serve as participants in a usability test. In order to test the guidance provided by the tutorial and quick start guides, three users will perform their tasks without these guides and three will be allowed to use these guides.
• Instruct each participant to use the site and the content (i.e., text & multi-media) that we will supply to perform a scripted set of typical user tasks.
• Observe and record participants progress and difficulties, if any.
• Participants will complete a survey on their satisfaction with using the site.
• Report on the results from all the participants' interactions with the site.

Strategy - Phase Two

• By uploading and converting various multi-media formats determine the best practices and potential problems in adding audio or video content to webshops and mini-courses.
• Assess whether Flash or any other format might be better suited to the needs of our audiences than other multi-media formats.
• Assess what procedures are the most appropriate for making PowerPoint presentations accessible (i.e., section 508 compliant).

Strategy - Phase Three

• Give the description of a pre-production workflow to usability test participants.
• Ask them to try the process with their own content.
• Collect and report on their responses to a survey on its use.

Objectives

• Identify any aspects of the Online Training section of the T/TAC Online system that could be improved.
• Provide best practices for selecting and utilizing media formats in webshops and mini-courses.

Levels of Evaluation

• Phase one will be a Level 1 & Level 3 evaluation.
• Phase two isn't covered by Kirkpatrick's levels of evaluation.
• Phase three will be a Level 1 evaluation.

Measurement Approaches

• Observers of participants during the usability testing will gather data on the time required for each task, the number & nature of errors committed, and the comments provided by participants.
• Data on time required to convert and upload various multi-media formats will be examined in determining best multi-media formats to employ.
• Survey responses will be collected from participants after they have examined and tried the pre-production workflow guidance that we will provide to them after the usability test.

Formality and Informality of Methods and Rationale

• Phase 1 will use comparisons of pre and post performance via numeric results for task times and error rates, as well as survey results. The availability of the tutorial and quick start guide will separate the pre & post groups. This phase will employ quantitative survey measures of satisfaction with using the site.
• Phase 2 will use numeric comparisons on times required to convert, upload, and download various media formats.
• Phase 3 will employ quantitative survey measures of satisfaction with the pre-production workflow guidance.

Societal and Cultural Factors

• Most end users (e.g., teachers, educational professionals) and all staff have internet access
• Staff have training and experience in developing some types of electronic training materials

Constrains

• Increasing requirements from the state & federal government (e.g., SOLs, NCLB) mean greater need for staff to provide training
• Reduced funding for education restricts travel of staff
• Staff do not have training in developing training for exclusively online delivery
• Internet access is low bandwidth for end users, perhaps for some staff

Desired Results (Information Sought from Evaluation)

This evaluation should answer these questions:

1. Have the interface enhancements successfully addressed the previous perception of system users that the system was difficult to use?
2. Can users easily determine what action they are to take and in what sequence, both in planning their webshops or mini-courses and in uploading them?
3. Does the tutorial effectively prepare users to build webshops or mini-courses?
4. Is the quick guide an adequate reference for the target population?
5. Are there other enhancements that could be made to the software interface to improve usability?
6. What are the best media formats and multi-media practices for this population of uses?