Perceptions
of Learning in an Online Synchronous Chat
Stacy Connors
William
Warrick
George
Mason University
Research Questions and Hypotheses
Rationale for the Study
More and more teachers
today are exploring the benefits of providing instructional opportunities
to students using online communication tools. Whether the whole class
is offered online or the Internet is simply used to augment a traditional
face-to-face class, there appears to be a growing interest in online
teaching and learning. The U.S. Department of Education’s National
Center for Educational Statistics (NCES), reported that the number
of distance education programs increased by 72 percent from 1994-95
to 1997-98 (Quality On the Line: Benchmarks for Success in Internet-Based
Distance Education, 2000). However since the quality of these new
programs can vary widely, education experts are telling teachers to
shop around and ask questions first. Not all online programs offer
teachers the right mix of collaboration and feedback (Weiner, 2001).
There are a number of benefits to offering classes online. Cereijo,
Young, and Wilheim (2001) studied students’ comments while taking
an online course. Their research showed that students reported the
advantages of CD/Web-based learning to be: convenience, flexibility,
and opportunity for enhanced learning. Conversely, students reported
concerns about isolation, the learning environment, and technological
problems to be significant disadvantages.
Unfortunately, many attempts
to promote teaching and learning online are less than successful.
There is a high dropout rate among students who begin online courses
and much express dissatisfaction with the courses. (http://www.edweek.com)
One reason for this could be that instructors have not given sufficient
thought to the appropriate use of online learning and have simply
attempted to ‘recreate’ the classroom experience in an
online environment. They fail to take into account the unique nature
of how students and teachers/instructors interact with one another
online.
While it is relatively
easy to distribute information to students using a variety of online
tools such as e-mail and the world wide web, it is much more difficult
to promote interaction between students and teacher and other students.
Interaction is one of the most important components of any learning
experience (Vygotsky, 1978). While teachers recognize that interaction
is a critical component of learning in the classroom, they find it
difficult to promote that type of interaction online using chat rooms
or discussion boards. The traditional rules governing interpersonal
communication in the classroom simply do not apply at a distance.
This study sought to investigate
a number of variables that may affect the learning of students in
an online course.
Research questions
- During a synchronous,
online discussion, does the presence of an online moderator have
an effect on the student’s perception of learning?
- During a synchronous,
online discussion, does years of teaching experience, categorized
by years, have an affect off the student’s perception of learning?
- Is there a relationship
between the student’s years of teaching experience and the
participant’s age to the student’s perception of learning?
Hypotheses
- Students engaged in
a synchronous, online discussion with a moderator would feel as
though more learning took place.
- Students with more
years teaching experience, engaged in a synchronous, online discussion
would feel as though more learning took place.
- The student’s
perception of learning can be predicted by the years of teaching
experience and the participant’s age.
Null Hypotheses
- For the synchronous
online learning context, there is no statistically significant difference
in the mean perception of learning scores for participants in the
two moderator groups.
- For the synchronous
online learning context, there is no statistically significant difference
in the mean perception of learning scores for participants, regardless
of their years of teaching experience.
- The years of teaching
experience and the participant’s age can not be used to predict
the student’s perception of learning.
Data Collection
Variables in the Study
The goal of this study
was to discover the effects of a number of variables on the learning
of participants in synchronous, online discussions. During the web
based learning course, students had the opportunity to participate
in two online, synchronous discussions. The discussions were designed
to integrate with the course in such a way as to provide a goal and
purpose for the discussion. Thus, during each of the discussions,
students had a particular learning goal. Given this framework, our
interest was in the effects of certain variables on the students’
quality of learning for each of the discussions.
Synchronous discussions
online are those where the participants are all interacting simultaneously
in real time. This type of communication, conducted over the internet
is often called ‘chat’. Participants in the Web-Based
Learning class logged on to the BlackBoard course delivery system
and entered the “Virtual Classroom” at a prearranged time.
The course syllabus provided the students with a topic to discuss
and a product to be submitted to the instructor following the chat.
There are three synchronous meetings scheduled for the semester. This
study looked at the first two. The first online discussion was moderated
by one of the instructors for Web-Based Learning. The role of the
moderator was to provide prompts to the participants, provide assistance
to students having difficulties, and maintain the focus of the group
on the topic.
For the first question,
the presence or absence of an online moderator (MOD) was examined
as an independent variable. As part of the Web-based Learning course,
students participated in two scheduled online discussions on topics
related to the course. During the first online discussion, a moderator
participated and was responsible for guiding the conversation. During
the second online discussion, no such moderator participated.
To answer the second question,
the number of years of teaching experience (TEACHCAT) was used as
the independent variable.
In answering the third
question, the students’ years of teaching experience and the
age of the participant (AGECAT) were the independent variables.
The dependent variable
in the study was the perception of the quality of learning (PERCEPT)
as reported by the participants during each of the synchronous discussions
Participants were surveyed after each of the discussions and were
asked to report their opinions as to the efficacy of the experience
in terms of quality of learning.
Description of Instruments
The data for this study
was collected using three researcher created surveys. The first survey
consisted of a number of questions designed to collect demographic
data on the subjects. This first survey was also used to collect information
regarding the participants’ perceptions of learning through
interactive discussions, both online and face-to-face. The researchers
developed a questionnaire on a word processing program to be distributed
electronically as described below. This questionnaire used a combination
of short answer and Likert scale questions. The survey was given to
several of the researchers’ coworkers who completed it and gave
feedback as to it’s ease of completion.
The second survey consisted
of four questions designed to elicit responses concerning the participants’
perceptions of learning quality of moderated online chats. This survey
was posted electronically at a commercial survey web site (http://www.zoomerang.com)
and the url was provided to the students who were then instructed
to access the site and respond to the survey. The questions in this
survey were all of a Likert scale design. There was some concern about
the ability of the participants to access and complete the survey
completely online. Discussions with the students and a response rate
greater than 95% indicated that the majority of students had no difficulty
at all in using the online survey.
The final survey that
was completed by the participants was, again, posted to the Zoomerang
web site. This survey consisted of five questions related to the students’
perception of learning quality in unmoderated online chats.
Each of the 59 students
enrolled in the Web-based learning course were surveyed for this study.
Data Collection Procedures
Since the participants
in the study were widely dispersed geographically, it was decided
that the most efficient method for distributing and collecting the
surveys was via e-mail. All of the students were sent a word-processed
file containing the survey. Their instruction was to save the file
then open it and answer each question on their computers. They were
then instructed to save the file and attach it in an e-mail to the
researchers’ faculty advisor, Dr. Priscilla Norton. Dr. Norton
then printed the surveys as they were returned and gave them to the
researchers. This provided a means whereby the anonymity of the participants
could be protected.
The participants were
surveyed prior to their first online discussion and then again after
the first discussion and, finally, after the second discussion.
All of the subjects are
participants in the program’s Web Based Learning Class (EDIT
797). During this class, all instruction and interaction between student
and instructor takes place online. For the purposes of the WBL class,
all cohorts are combined into one large class with no distinctions
based on location.
Coding and Organization
of Data
SPSS – Statistical
Package for Social Sciences, Version 10.0 was use to tabulate data.
Variables were coded as
follows:
MOD: nominal, 1=moderated
2=unmoderated
AGECAT: scale, participant age recoded into categories: 1 = 0-27 yrs.;
2 = 28-34; 3 = 35-highest
TEACHCAT: scale, 1 = 0-4 yrs.; 2 = 5-8 yrs.; 3 = 9-highest
PERCEPT: ratio, student rating of the quality of the learning experience
on a scale from 1 to 5
Data Analysis
Rationale for Analysis Technique
SPSS: Statistical Package for the Social Sciences, Version 10.0 (Mac)
was used to run all procedures. Descriptive statistics were obtained
to describe the sample including the sample size, measures of central
tendency (mean, median and mode), and measures of dispersion (range,
variance, standard deviation, quartile ranges, and minimum and maximum
values).
Table 1 shows the variables
that were used. Those variables are: presence or absence of moderator,
participant’s age, years of teaching experience, perception
of learning ratio.
Table
1: Descriptive Statistics for Variables Related to ANOVA and MLR
A two-way analysis of
variance (ANOVA) was used to compare differences in the means between
the presence of an online mentor and years of teaching experience
and their effect on student’s perception of learning. Post hoc
procedures were used to test the data for the underlying assumption
of analysis of variance.
A multiple linear regression (MLR) procedure was used to determine
if the perception of learning can be predicted by years of teaching
experience and number of students in an online, synchronous discussion.
Additional procedures were used to determine if the data will meet
the assumptions underlying multiple linear regression.
Exploratory Analysis
Exploratory analyses were
conducted to determine if the assumptions for both the ANOVA and MLR
were met. The alpha level was set to .05 for all tests of the assumptions.
Three assumptions for
the analysis of variance were tested for this study. These assumptions
relate to: independence and randomness, normality, and equal variances.
The sample for this study
meets the first assumption, independence. The observations within
each group are independent. Subjects from this study were not randomly
selected from the whole population of online learners. This study
would be considered a convenience sample because all the subjects
are members of an online class that is currently being taught. It
is also assumed that the sample came from a normal population of students
in web-based learning class.
In order to test for normality,
the Kolmogorov-Smirnov Test was used. It assessed the distribution
of scores associated with the dependent variable, perception of learning.
Q-Q plots were used to provide a visual representation of the data.
The results of the Kolmogorov-Smirnov Test, shown in Tables 2 and
3, indicate that the distribution of scores of perception of learning
for the two groups, moderated and unmoderated and within the recoded
years of teaching experience are not normally distributed when p is
set at .05. We know the scores are not normally distributed because
the significance was less than .05 therefore we rejected the null
hypotheses in favor of the alternative and the distribution is not
normal. Q-Q plots, provided in Appendix A, also support that the scores
are not normally distributed as the points did not fall on a straight
line.
Table
2: Test of Normality for Moderated or Unmoderated Chat
Table
3: Test of Normality for Years of Teaching Experience
The Levene test of Homogeneity
of Variance was used to test the final assumption of equal variance
for both independent variables. This assumption indicates that, across
the groups, the variance of the dependent measure are equal. We know
the variances for the population are not equal because the significance
was less than .05 therefore we rejected the null hypotheses in favor
of the alternative and the variances are not equal. Test statistics
were shown here in Tables 4 and 5. Box plots were used to inspect
this data and are included in Appendix B.
Table
4: Test of Homogeneity of Variance: Recoded Ages
Table
5: Tests of Homogeneity of Variance: Years of Teaching Experience
There are four assumptions
that must be met for multiple linear regression. They are independent
observations for each value of the independent variable, the distribution
of the values of the dependent variable must be normal, the variance
of the distribution of the dependent variable must be the same for
all values of the independent variable (known as homoscedasticity)
and the relationship between the dependent and the independent variable
must be linear in the population.
Independent observation of pairs can be assumed because the participants
were randomly assigned to the groups for the purposes of online synchronous
discussion. None of the participants was a member of more than one
of the groups at the same time. No observations influenced any other
observations.
The studentized deleted residuals were used to test the second assumption
of normality. The results of the Kolmogorov-Smirnov test, shown in
Table 6, indicate that the distribution of scores of perception of
learning for each value of the independent variables, MOD and TEACHCAT
are not normally distributed when p is set at .05. The corresponding
stem and leaf plots and Q-Q plots that are shown in Appendix C, also
prove that the variables are not normally distributed.
Table 6: MLR Test for Normality
The third assumption,
homogeneity of variances, was tested by plotting the studentized residuals
against the predicted values. Since the scatterplot shown in Figure
1 presents a particular pattern, the variance is considered to not
be constant. Attempts were made to transform the data by taking the
square root, by squaring, and by taking the natural log of the dependent
variable. All attempts yielded the same output: the variance appeared
not to be constant.
Figure 1: Testing for Constant Variance
The fourth and final assumption is that the relationship between the
two variables must be linear. To begin, a scatterplot matrix was constructed
with the the dependent variable and the two independent variables.
Figure 2 indicates that there was no linear relationship between the
variables. After transforming the independent variables using the
square root, square, and natural log, no linear relationship was found.
Therefore this data does not meet this final assumption of linear
relationships.
Figure 2: Scatterplot Matrix of Linear Relationships
Secondly, each independent
variable was plotted against the studentized deleted residuals and
did not indicate a strong linear relationship. Figures 3 and 4 present
the scatterplot.
Figure 3: Test for Linear Relationship Studentized Deleted Residual,
TEACHCAT
Figure 4: Test for Linear Relationship Studentized Deleted Residual,
AGECAT
In conclusion, while every effort was made to meet the assumptions
of the analysis of variance and multiple linear regression by transformation
of the data, all attempts failed. This poses a significant threat
to the validity of our findings.
Description of Analyses
Related to Research Questions
A two-way analysis of
variance procedure was used to answer the research questions: 1) During
a synchronous, online discussion, does the presence of an online moderator
have an effect on the student’s perception of learning? 2) During
a synchronous, online discussion, does years of teaching experience,
categorized by years, have an affect on the student’s perception
of learning?
The multiple linear regression
procedure was used to answer the research question: Is there a relationship
between the student’s years of teaching experience and the participant’s
age and the student’s perception of learning?
Research
Design
Description of the Sample
The participants in this
study were all students in a Web-based Learning course taught as part
of a Master’s degree in Education program. The participants
range in age from 23 to 54 (mean= 34) and in years of teaching experience
from 1 to 32 (mean = 7.8). Three sections were studied with a combined
total of 59 students. Students in these three sections of the class
were randomly assigned to participate in either moderated discussion
groups (N=30) or unmoderated discussion groups (N=29).
Students in this study
are generally representative of students engaged in postgraduate,
professional study. Furthermore, this sample is representative of
the normal population of teachers who seek an advanced degree in education
and participate in an interactive online course.
Description of the Research
Design
This study is considered
a experimental study. The participants in the study were randomly
assigned to either a moderated or unmoderated chat group The selection
of the sample is not random since all of the participants in the Web-Based
Learning class were a part of the study. The participants were divided
into two groups based upon the type of online synchronous chat they
participated in. The participants were surveyed prior to the start
of the course, at the conclusion of the first online, synchronous
chat, and upon completion of the final online chat. The surveys were
designed to collect demographic data on the participants as well as
their attitudes, beliefs, and perception of learning in online chats.
Design Validity
Perhaps the greatest threat
to validity in this study is the method of delivery for the survey.
In each of the three instances that the survey was distributed, it
was done electronically. Thus, it is possible that, in collecting
the data from the participants there was the risk of incorrect data
being submitted due to problems with Internet access by the participant,
the facility with which the participant uses the world wide web and/or
a word processor, and the accessibility of the web page on which the
survey was posted.
Statistical Validity
All statistical procedures
performed used an alpha level of .05. All testing for both ANOVA and
MLR failed to meet the assumptions.
Results
Descriptive Findings
Descriptive statistics
were computed using SPSS for several variables for both the ANOVA
and MLR procedures. Data for the descriptive statistics are shown
in Table 7.
Out of the 59 participants
in the study, 49% (N=29) of participants did their synchronous chats
without a moderator (unmoderated) while 51% of participants had a
moderator participate in their synchronous chats.
For the MLR analysis, the age of the participant as well as the years
of teaching experience were examined. The mean age of the participants
was 34 years with a standard deviation of 8.72 and a variance of 76.172.
The range in age was from 23 years to 54 years. Additionally, the
years of teaching experience on the part of the participants was surveyed.
The survey asking for demographic data asked only that the participants
indicate the number of whole years of experience in teaching they
had. The mean number of years of teaching experience was 7.88 years
with a standard deviation of 6.32 and a variance of 39.97. The number
of years of teaching experience ranged from one year to 32 years.
Table 7 displays the descriptive
statistics for the ANOVA procedure. The mean rating for usefulness
of online chat for those teachers with 0-4 years of teaching experience
participating in moderated discussions is 3.43 (s=.78). Teachers having
between five and eight years of experience had a mean usefulness rating
of 3.46 (s=.87) and teacher with nine or more years of teaching experience
rated the usefulness of their moderated online chat at a mean of 3.70
(s=1.06). Teachers who participated in unmoderated chats with 0-4
years of experience teaching had a mean usefulness rating of 3.4 (s=9.67).
Those teachers with between five and eight years of experience gave
the unmoderated chat a mean usefulness rating of 3.5 (s=1.18). Finally,
teachers with nine or more years of teaching experience rated the
usefulness of their unmoderated chat at a mean of 4 (s=0.0).
Table
7: Descriptive Statistics for ANOVA
Inferential Findings
A two-way ANOVA was conducted
to determine the effects of a moderated chat and years of teaching
experiences and their interaction with perception of learning. Between
moderated and unmoderated chats, there is no significant difference
in the mean number of the perception of learning. F(1,53)=.184, p=.670.
The means for moderated and unmoderated, respectively were 3.53 and
3.62. Null is not rejected. Among various categories of teaching experience
there is no significant difference in the mean number of perception
of learning. F(2,53)=1.24, p=.298. Null is not rejected. There is
no significant interaction between moderated chats and years of teaching
categories for the mean number of perception of learning. F(2,53)=.168,
p=.846. Null is not rejected. Therefore, it was not possible to reject
the null hypothesis that there will be no significant difference in
the mean perception of learning scores for participants in the two
moderated groups, nor their years of teaching experience. The results
of the ANOVA are shown in Table 8.
Table
8: ANOVA Test of Between-Subjects Effects
A multiple linear regression was performed to determine if years of
teaching experience and age were good predictors of perception of
learning. The results of the regression are displayed in Tables 9,
10, and 11.
Table
9: Multiple Linear Regression: Model Summary
Table
10: Multiple Linear Regression: ANOVA
Table
11: Multiple Linear Regression: Coefficients
The best model for predicting
perception of learning scores in this sample involved both the predictor
variables: AGECAT and TEACHCAT. The model accounted for 7.3% of the
variance, R2=.073. The prediction equation can be constructed using
the unstandardized coefficients (B) shown in Table 11: perception
of learning = 3.020 + -.039(TEACHCAT) + .329(AGECAT). Because the
significance of the regression is greater than .05, F=(2,58) = 2.221,
p=.118, in ANOVA results that indicate that it is not possible to
reject the null hypothesis that all of the predictors scores are equal
to zero, thus indicating that the model was not an adequate prediction
model.
Conclusions
Interpretations
The results of this study
were somewhat disappointing. Although any results can be viewed constructively
to some degree, it would have been nice to yield results with some
type of significance.
In regard to the first
hypothesis, students engaged in a synchronous, online discussion with
a moderator would feel as though more learning took place, the study
failed to provide any statistical proof that a relationship existed.
The results of the ANOVA tests indicate that there is no relationship
between the independent and dependent variables. We failed to reject
the null hypothesis that there is no significant differences in the
perception of learning scores in relation to the chat being moderated
or unmoderated.
The results regarding
this variable were the most surprising. Although one would hope that
students have the ability to learn either with or without a moderator,
we expected to find a greater degree of variation in the perception
of learning between moderated and unmoderated chats. Indeed, our feeling
was that students in moderated chats would report a higher degree
of usefulness for chats. One possible explanation for this might be
the order in which the chats took place. Only the first synchronous
chat had a moderator. Although the students felt as though they learned
in the first chat with the moderator, they also learned in the unmoderated
chats. These results may be explained by the student’s comfort
level with the medium or simply that they were further along in the
semester. It could also be that the students began to understand the
objectives of the course, had proper modeling by the moderator in
the first chat, and may have transferred those skills to the subsequent
chats that were unmoderated which may have contributed to their overall
perception of learning.
In regard to the second hypothesis, students with more years teaching
experience, engaged in a synchronous, online discussion would feel
as though more learning took place, proved to be incorrect as well.
We failed to reject the null hypothesis that there is no significant
difference in the mean perception of learning scores regardless of
their years of teaching experience. In this instance, the hypothesis
that participants with more years of teaching experience would report
a greater perception of learning was based on our notion that the
more experienced teacher would be better able to integrate the online
chat and utilize in for learning to a greater degree than the less
experienced teacher. It is possible that the concept of virtual chats
is still new enough to all groups that they report the same level
of perception of learning. It is also possible that the relatively
short exposure that all of the teachers had in this study was not
sufficient to illustrate the potential of virtual chatting and thus
all experience groups perceived the same level of usefulness.
Not only was there no
significant data found to support either of the hypotheses, the study
failed to meet any of the assumptions of the ANOVA.
In regard to the multiple
linear regression, not only did the researchers fail to reject the
null hypothesis, that the years of teaching experience and the participants’
age cannot be used to predict the students perception of learning,
the data also failed to meet the assumptions for the multiple linear
regression.
In looking at the factors
of this study, it was difficult to actually have the students rate
their own learning in a survey. There are a number of factors which
may explain this. First, it is more difficult for students to rate
their own learning when the learning goals are not specifically observable
and tangible. In this study, the goals for the first chat were simply
to discuss the topic that was given and provide the instructor with
a synthesis of that discussion. Second, there may be a skewing of
the reported learning perceptions due to the fact that the researchers
in this study were also the instructors. Students may have felt, even
subconsciously, that they should rate their learning higher because
of this
Implications
One serious flaw in this
study is the fact that the online class was not a variable in the
study. Given that the notion of learning online is still relatively
new, we take for granted that not all students have an equal comfort
level in this medium. It is very important to look at other factors
that influence learning. Some of those factors may be learning styles,
comfort regarding the topic studied, background knowledge, or in the
case of an online learning environment, comfort level regarding the
use of technology.
Online learning is still
an area in need of more research. The results of this study simply
inspire us to continue to dig deeper into the area of online learning
and search for ways to create environments in which students can learn.
Recommendations
This study failed to illuminate
factors that might influence the rate or quality of learning in online,
synchronous chats. It appears to show that the presence of a moderator
does not effect the learning of the participants. The reasons for
this are discussed elsewhere in this paper, but, it does appear that
further investigation into the role of a moderator may be warranted.
It would be useful, too, to measure learning in the moderated and
unmoderated environments with a more objective instrument than self-reporting.
There are certainly factors
that influence the success of online, synchronous chats as a learning
tool. A good topic for further study might be to delineate some of
the variables that might influence the success of online chats and
study their effect on learning.
As the use of such tools
becomes more prevalent, it is imperative that they be implemented
in the most effective manner possible.
Appendix A: ANOVA Tests
of Normality: Q-Q Plots
Figure 1: Q-Q Plot for
Moderated Chat
