“Portfolio Assessment in the Graduate School
of Education:
Process, Validity, and Reliability”
William Warrick
EDIT 797 - Technology and Assessment
Dr. Anthony Kelley
June 22, 2002
Educators everywhere must, at various times, assess students. At every
grade, in every subject, teachers take measure of the learning of their
students in order to make decisions about their teaching and the future
direction for learning of their classes. With all forms of assessment,
educators are searching for indicators of their students’ learning,
the efficacy of the curriculum with respect to standards, and ways in
which they can improve the overall learning and development of their
students. With increases both in class sizes and the amount of curriculum
to cover and the need to make wise use of limited time and resources,
the choice of assessment tools is critical. New research on learning
and assessment has provided teachers with a variety of ways to view
assessment and the role of the student in evaluation.
The assessment of students can take place in a number
of ways. Paper and pencil tests, long the standard for classroom testing,
can take on a number of forms. More often than not, multiple choice,
true-false, and short answer tests often represent the most economical
and expedient choice for assessing the attainment of learning targets.
The strength of these tools lies in their structure and generalizability
to other areas. Properly constructed tests can provide valid and reliable
measures to be used by decision makers.
Other forms of assessment rely on the students’
performance on tasks that are given by the teacher. Teachers assign
tasks and then, using criteria outlined in a rubric, assess the students’
performance on the task. This type of assessment has the advantage of
giving the student situations in which to demonstrate both skills and
cognition in a particular domain.
Increasingly, teachers are using authentic and alternate
forms of assessment in which the learner has a more active role in the
assessment process. Nitko (2001) suggests that the benefit of these
types of authentic or performance assessments is that they provide students
with the chance to apply their learning to specific realistic situations.
Portfolio assessment is gaining acceptance in a wide range
of educational settings. Where once the term portfolio evoked images
of artists and large leather cases holding drawings and works of art,
portfolios today have come to mean a collection of products, generally
produced by the student, which, together, represent an integrated view
of the student’s learning. The works contained in a portfolio
are selected to illustrate examples of the student’s work or progress
in a domain. The use of the portfolio as an assessment tool represents
a shift from, and perhaps a reaction to, standardized testing. These
types of assessment are more concerned with providing teachers with
ranking of students against national or state norms or criteria. Portfolios
are not as much concerned with grading students as with assessing learning
(Barton & Collins, 1997).
This paper seeks to analyze an implementation of the
portfolio assessment process at George Mason University. The portfolio
assessments will be described in the context of both traditional paper
and binder portfolios as well as electronically published portfolios.
Nitko (2001) defines portfolios as “a limited collection
of a student’s work”. Rather than a collection of everything
that a student has produced, it is a representative sample of the types
of work which illustrate a student’s attainment of learning targets
or growth over time. The uses of portfolios as an assessment tool are
varied, depending on the learning goal, but there are two main types
of portfolios in assessment - best work portfolios and learning and
growth portfolios. Best work portfolios are those which showcase (literally,
at times) a student’s best work. Learning and growth portfolios
are designed to be works in progress over the duration of a student’s
studies. Portfolios often serve a dual purpose. Barton and Collins (1997)
illustrate how educational portfolios borrow attributes from art and
finance portfolios. Where the artist’s portfolio will contain
his or her best works, and the financier’s contain an ongoing
record of transactions, an educational portfolio takes elements from
both. Educational portfolios, then, can demonstrate a student’s
best work while showing progress and integrative thinking over time.
The process involved in developing a portfolio for assessment
varies according to the learning goals to be addressed and the purposes
of the assessment. Common to most, however, is the following:
- The learning goals to be assessed are articulated
- The student collects all products.
- The student selects those artifacts which provide evidence of accomplishing
the major learning targets.
- The student adds a review or reflection on each item or on the portfolio
as a whole.
- The portfolio is assembled.
- The student and teacher(s) meet to review the portfolio and provide
the assessment.
Unlike many traditional forms of assessment, the act
of creating the portfolio is a part of the learning process. In selecting
items to be included in the portfolio, the student critically examines
all possible products which could be included. On the basis of an items
relevance to the learning goals to be assessed, or following direction
from the teacher, artifacts are selected. Nitko (2001) suggests that
as students get older they can assume more responsibility for selecting
items which will comprise the portfolio. The process of selection affords
the student the opportunity to review and reflect on the work of a semester
or course of study, a metacognitive exercise which reinforces their
learning.
A critical component of the portfolio as an assessment
tool is the reflective nature of the process. A normal practice when
using portfolios is to have the student provide an explanation and rationale
for the inclusion of the various items. This part of the reflective
process reinforces the learning and understanding. The very act of gathering
and choosing artifacts for inclusion gives the student the opportunity
to reflect on the learning targets set by the teacher.
Once the student has selected all of the material to
be included in the portfolio, they must compile it in such a way as
to illustrate their growth and learning. The organization of a number
of artifacts to represent learning growth over time allows both the
teacher and the student to make connections among the various subjects/courses
the student has taken. From these connections, the student is positioned
to integrate and, perhaps more importantly, articulate a perception
of his or her learning into a whole.
Once the portfolio is complete, it is submitted to the
teacher for review and evaluation. In some cases, the portfolio is presented
to the teacher(s) and the process includes a dialogue between the rater
and the student. The student presents the portfolio and gives explanations
for it’s parts. The teacher (or committee of teachers) may then
question the student to elicit verbal evidence of the student’s
learning. The teacher or committee will be able to offer guidance to
the student if deficiencies are noted. Portfolios help teachers negotiate
the conflict between the role of supportive, welcoming helper and the
role of critical, skeptical, evaluator (Elbow and Belanoff ,1997). Teachers
can also take this opportunity to obtain feedback on their own teaching.
A review of the portfolio provides the teacher and student the opportunity
to discuss the connections.
With the advent of web publishing and advances and accessibility
of digital media tools, many portfolios are now being created electronically
and published on the web. Among other things, this makes the portfolio
available to potential evaluators or employers at any time. The electronic
portfolio is well suited to modifications and additions as the student
moves through a program. Additionally, portfolios published on the web
can provide links to sites containing background and supporting information
to better illustrate the student’s work.
The web-based or electronic portfolio also allows the
student to include other types of media such as graphics, animations,
sound, and video. These further enhance the functionality of the tool
and provide the student with greater flexibility in deciding the products
to include. Using a multimedia format also allows for interactive evidence
to be included (Thomas, Lamson, & King, 2001)
One other critical benefit of an electronic portfolio
is that the creator of the work can create links between and among the
various artifacts. These links can serve to illustrate the integrative
nature of the learning that has occurred. One of the most alluring aspects
of hypertext is the ability to imitate the associative connections natural
to the workings of the human brain (Fischer, 1997). Simulating the way
in which we organize and retrieve information in our minds, the electronic
portfolio links related ‘chunks’ of information together.
This allows the student to produce a portfolio unfettered by the linearity
of print media. The portfolio can be organized to support and promote
connections to its various parts and presented in such a way as to allow,
even encourage, the reader to follow connections. The student’s
organization of the portfolio, then, becomes a metacognitive exercise
for the student. Nitko (2001) warns, though, that self-reflection is
a mental activity and the assessment of that activity must be indirect.
For graduate school students, the goals of assessment
take on a different aspect. While there are skills and concepts to be
mastered, the emphasis is on reflection and analysis of the learning
process. Many graduate schools - whether by their own design or through
mandates for accreditation - have adopted the portfolio as a form of
assessment for their licensure candidates.
In the late 1990’s, George Mason University began
to offer students the option of creating a learning and growth portfolio
in lieu of the traditional comprehensive examination. This comprehensive
examination was administered at the conclusion of a student’s
coursework and used to assess the candidate’s body of knowledge
in a field. The results of this examination were used to advanced the
student to candidacy in the Ph.D. program. There were those, however,
who felt that a multi-hour comprehensive written examination was an
artificial determinator of learning experiences because the conditions
of the exam were unlike how people usually work in the field (Heiges,
1997).
It is now a requirement for all Ph.D. students at George
Mason’s Graduate School of Education to create and maintain a
portfolio to illustrate and present their learning and growth. The guidelines
provided to students are as follows (Graduate School of Education, 1997):
The Ph.D. in Education Portfolio is an organized, yet
selective collection of documents designed to facilitate a student’s
academic and professional development and to provide a basis for evaluating
degree progress. The portfolio represents the scope and depth of a
student’s goals, plans, and accomplishments in coursework, independent
study, research, internships, and other advanced learning activities.
The portfolio thus provides both a vehicle for self-reflection and
a comprehensive record of a doctoral student’s experiences and
ongoing progress toward his or her academic and professional goals.
Drawing on the benefits of portfolio assessments described
earlier, the Ph.D. portfolio represents the the student’s growth
and development throughout their coursework. The individual artifacts
are not assessed in and of themselves. These items have been assessed
and graded within the individual courses for which they have been produced.
Rather, the assessment of the portfolio for the doctoral student focuses
on the cumulative nature of the products. The whole of the portfolio
is taken as a representation of the student’s integrated learning
over their program. The portfolio is developed to show how the student
sees the integration of his or her coursework and its relation to Ph.D.
candidacy.
The Ph. D. student at George Mason University is involved
in three portfolio reviews prior to advancement to candidacy. The first
two reviews are scheduled after 15 and 30 credit hours have been taken.
(or, at the 4th and 7th semesters of study, whichever comes first).
The first and second portfolio reviews are designed to be face-to-face
meetings with the students doctoral advising committee. This committee
is usually made up of faculty members who share a background in the
content or research area in which the student plans to study.
The periodic portfolio reviews are an opportunity for
the student to share his portfolio and thus his view of his place in
the Ph. D. program. The committee reviews the student’s goals,
plans, and accomplishments, and discusses possible modifications and
additional work needed to proceed in the doctoral program.
During the final portfolio review, the student, the focus
shifts from academic and professional development to formal evaluation.
The purpose of the final review is to assess the student’s readiness
to proceed to the dissertation phase of the program. At the conclusion
of the review, the student is asked to leave and the committee discusses
the student’s progress and consensus is reached on one of three
possible outcomes (Graduate School of Education, 1997):
- Student has demonstrated readiness to proceed to the dissertation
phase of the program (non-binding recommendations for modifications
or additional work may be specified)
- Student may proceed to the dissertation phase of the program when
required actions are completed; a second meeting is not required
- A second comprehensive assessment meeting will be scheduled when
required actions are completed.
The Graduate School of Education, on its web site, provides
a sample table of contents listing the required elements of the portfolio.
Inasmuch as the portfolio for Ph.D. students is ultimately
used to advance students to candidacy for their degree, the validity
and reliability of the results of the portfolio review must be taken
into consideration. The portfolio assessment takes the place of the
traditional comprehensive exams and, to the degree possible, the outcome
should be shown to have some correlation both to the assessment that
it replaces and to the candidate’s success once they have advanced
to that stage. While a sound validity argument is relevant to any assessment
purpose, it is crucial when decisions or actions based on the assessment
have significant consequences for candidates (Moss 1998). Given the
high stakes nature of the comprehensive review of the Ph.D. portfolio,
it is well that we consider the tool in light of it’s validity
and reliability. Nitko (2001) lists eight types of validity evidence
for educational assessments. Each of the eight types of validity evidence
and their influence on the portfolio are discussed here:
1. Content Evidence
Content validity refers to the degree to which the assessment
tool represents the domain of important content. In this area, the results
of a portfolio assessment fare well due to the fact that, by its very
nature, the tool must represent the content. The portfolio that is created
by the student is based upon his or her unique goals in the Ph.D. program.
These goals are the result of working with the advisor in the program.
Thus, the students’ selection of artifacts is based wholly on
their own accomplishments and goals. Since this is unique to each individual
student, there can be no question as to the validity of the content
which is represented. It would be unrealistic to compare the portfolio
contents, in terms of representativeness, with other students’
collections because of the unique nature of the program of study. Therefore,
we must compare the student’s portfolio with the committee’s
notion of the content which represents the whole of the student’s
experience.
What remains to be judged, then, is the reflective portion
of the portfolio. This area, showing the candidate’s thinking
and reflection on his or her coursework and how it integrates together,
is critical to the assessment process. The reflections included in the
portfolio show the review committee the depth of the student’s
understanding of how the coursework is integrated together to form a
whole. The products included in the portfolio have been assessed by
the individual professors in classes. What remains to be assessed is
how the student integrates all of the learning together to show their
understanding of their unique program.
2. Substantive Evidence
The portfolio process at George Mason is heavily dependent
upon thinking processes and skills. From the moment the Ph.D. student
begins work on a portfolio, these thinking skills come into play. The
student must, from a collection of all of their work, evaluate those
products which should and should not be included in the portfolio. This
critical analysis of artifacts forces the student to judge the appropriateness
of the artifact in terms of his or her own learning goals as well as
the goals of the program. Once items are selected for inclusion in the
student’s portfolio, the student creates a reflection on those
artifacts. The purpose of the reflection is twofold. First, the student
reflects on the meaning of the individual product itself - what the
product or artifact taught the student in the class, the processes involved
in creating the artifact, and the degree to which the artifact represents
the attainment of a learning goal for that class. Second, the student
must ‘place’ the artifact in relation to their entire program
- How does this product ‘fit’ with the larger goals of the
Ph.D. program? How does this piece interact with products from other
classes to create knowledge of a larger scope? Thinking skills are further
evidenced during the portfolio review process in that the student must
develop an argument for the inclusion of the particular products they
have chosen.
The portfolio evaluation in place at George Mason University
not only necessitates the use of the types of thinking skills that demonstrate
the use of the types of thinking skills required for advancement to
candidacy, they also give evidence to the types of thinking skills the
candidate will use in further study and professional life. The results
of a portfolio review are thus validated by this characteristic.
3. Internal Structure Evidence
As described earlier in this paper, the complete portfolio
creation and review process involves a number of steps. While the product
is reviewed and assessed holistically, the individual tasks leading
to the whole are critical to the assessment process.
The portfolio review and assessment process can be broken
down into three distinct parts: 1) the process of selecting artifacts
for inclusion; 2) organizing the entries; 3) reflection on the items
specifically and the portfolio and process as a whole. Individually,
these processes influence the validity of using the results of the portfolio
review. The raters of the portfolio, typically the members of the student’s
program committee, must be aware of the processes involved in portfolio
development and their final evaluation must be based upon their assessment
of those parts.
4. External Structure Evidence
The validity of any assessment relies on how well the
results of the assessment correlate with other variables or criteria
(Nitko, 2001). The types of evidence of correlation depend on the interpretation
and use of the assessment. In the case of the portfolio assessment as
a condition of advancement to candidacy, we would expect there to be
a positive correlation between the results of the portfolio review process
and successful completion of the Ph.D. program. Nitko allows for hypothesizing
about he relationships of portfolio results to external criteria on
the basis of experience and logical analysis.
The results of the portfolio review process are related
to predictive validity evidence (Nitko, 2001). The future performance
of the Ph.D. student should be evident in the results of the review.
In many ways, the results of the view point to concurrent validity evidence,
or the current position of the student in terms of his preparation for
advancement.
5. Reliability Evidence
Reliability, or the consistency of assessment results
over time or between raters, is necessary to insure validity of an evaluative
instrument. Most significant in the portfolio review process is inter-rater
reliability. When scorers have only one sample of something, created
under certain conditions, they have a much easier time agreeing with
each other than when they score the mixture of pieces in a portfolio
(Elbow and Belanoff, 1997).
Elbow and Belanoff (1997) suggest that one way to alleviate
the conflicting views between raters of a portfolio is not to attempt
to critically score all of the portfolios. Since most raters will agree
on those portfolios that do not meet advancement criteria and, similarly,
most will agree on those that are superior, they suggest that the best
and the worst portfolios should be scored and not the remainder. In
fact, they argue, if the assessment is to be used as an exit or placement
test, only the poor portfolios need to be identified. They call this
process minimal or limited holistic scoring. While this seems heretical
at first read, it makes sense. There is no gradation in the scores or
placement of the Ph.D. students when they advance to candidacy at George
Mason. They either advance, or not. The time and effort required to
give scores or ratings to all student’s portfolios can be regained
by providing interaction opportunities to students. This is not to suggest
that those students whose portfolios fall somewhere between excellent
and unsatisfactory are not to benefit from feedback or interaction with
their committee. On the contrary, once the portfolio is accepted as
meeting the requirements, the student and the committee are able to
concentrate on a deeper analysis of the portfolio’s contents and
the student’s understanding. Raters using minimal holistic scoring
are less concerned with scores than with differences found within the
portfolio and can concentrate comments and feedback on them (Elbow &
Belanoff, 1997). Inasmuch as the GMU portfolio review process does not
provide students with a score, this minimal holistic scoring system
appears to be supported.
6. Generalization Evidence
Using portfolios as an assessment tool necessitates that
the rater as well as the student be aware of a number of factors which
may alter a rater’s scoring of the portfolio. Among these, Nitko
(2001) lists the sophistication of the students. While generally this
refers to the degree that the students are “test-wise” or
have good test-taking strategies, in the case of the electronic portfolio
it could suggest a difference in the technical abilities of the student
to produce web-based portfolios that are technically sophisticated yet
no more relevant (in terms of content, reflections, and integrated thinking)
than a student who creates a simple web site. Indeed, students who put
more effort into making the portfolio attractive at the expense of its
representativeness, might stand to influence the rater’s decisions.
This is referred to as a “Halo effect”. Halo effect, according
to Nitko (2001) refers to the influence one characteristic of a person
(such as general impressions of a student) on another characteristic.
The validity issue here, obviously, is that the raters must guard against
rating one portfolio higher than another simply because of its appearance
or construction.
7. Consequential Evidence
The use of the portfolio review process implemented in
the Graduate School of Education has a number of consequences. Clearly,
the major consequence is that a prospective Ph. D. student is either
advance to candidacy or denied that rank. The benefits, though, of this
use of the portfolio results is that all students - those who advance
and those who do not - are given the opportunity to interact with the
members of the advisory committee and receive constructive feedback
about any deficiencies that are noted.
8. Practicality Evidence
Certainly, a paper and binder portfolio assessment tool
does not impact greatly on the budget of George Mason University. The
factors involved here would include the time of the members of the committee
spent on the review process and the cost to the university of hosting
the electronic portfolios on servers. The latter cost is not one to
be ignored. With new software and techniques available to facilitate
the creation of web-based portfolios and the increasing numbers of students
taking advantage of the resources offered (albeit grudgingly) by the
university, there are economic concerns with electronic portfolios.
However, the benefits of the assessment tool outweigh the potential
expense of providing storage space.
The software needed to create an electronic portfolio
is easily obtained and, more often than not, very inexpensive or free.
The tools available also do not require a great deal of investment in
time or money for training. With very little technical knowledge a person
can produce a very acceptable web page with Netscape Composer (Aschermann,
1999). Netscape Composer is free and is generally installed on most
university computers.
A number of teachers and students are reluctant to use
portfolios for assessing learning. Many teachers may employ a portfolio
assessment process but, because of its complexity may use it incorrectly
(Nitko, 2001). This is a practicality issue that may or may not influence
validity Obviously, teachers and students who wish to employ portfolios
as part of the assessment process should obtain training and experiences
in doing so.
Teachers concerned more with a ‘bottom-line’
score with which to place students on a linear scale with respect to
peers may not see the benefits of assessment by portfolio. The tool
simply does not lend itself well to the kind of rankings afforded by
standardized testing. However, teachers interested in gauging the integrated
learning of their students would find that the portfolio assessment
fits their needs.
The Graduate School of Education at George Mason University
recognizes the value of the portfolio as a tool to assess the learning,
growth, and direction of doctoral students. The process by which the
portfolio is reviewed gives the student and the student’s committee
the opportunity to interact with each other and the portfolio. This
negotiated learning is a critical component of the process. Without
it, the portfolio reverts to being simply a collection of artifacts.
It becomes inert knowledge. However, when judgments emerge from a thoughtful
process grounded in a multifaceted body of evidence and multiple points
of view, they are more likely to be valid and lead to continuing improvement
rather than just a stamp of approval or disapproval (Kimball & Haley,
1998). This, then, is the basis for the portfolio process at George
Mason.
I have had the opportunity on a number of occasions to
participate in the portfolio assessment process. First, as a student,
I found that I was in the unfamiliar position of being asked to personally
assess the work that I had completed and to discuss my view of how my
work served to inform my practice as an educator as an administrator.
I confess that my first foray into the world of creating a portfolio
was less than illuminating - either for myself or my instructors! I
soon became aware, however, of the goals of self-reflection and assessment
and approached the task each semester with greater motivation and interest.
Now that I find myself in the doctoral program at George Mason University,
I am again involved in the development of my required portfolio. I view
it now, however, with anticipation rather than trepidation. As a student,
I now look forward to being able to collect my thoughts and products
and organize them in such a way as to represent me and my way of viewing
my learning and my practice.
As a faculty member at George Mason University, I have
been able to participate in the review of students’ portfolios.
Here, too, despite my experiences as a student, I found it somewhat
disconcerting to assess a semester of work on the basis of the binders
handed to me. In reading my students’ reflections and discussing
their meaning (with the students themselves and another faculty member
with whom I taught), I came to see from the instructors point of view,
the benefit of this type of tool. It not only informed me of the student’s
learning, but also served to inform me of my own teaching as well.
As an alternative to a traditional paper and pencil examination, a portfolio
gives the student the means to be more of a participant in their learning
and assessment. The task of creating and presenting a portfolio of learning
involves the student in a rich process of self-reflection and analysis
of not only what they have learned, but how the learning goals are related
and how they promote success in future studies.
There are those who reject the notion that this form
of assessment on the basis of it’s seemingly informal and unique
form. However, when care is taken to address the issues involved with
portfolio assessment, the tool can be used validly and reliably in many
areas of instructional assessment.
References
Aschermann, J. R. (1999). Electronic portfolios: Why?
What? How? In: SITE 99: Society for Information Technology & Teacher
Education International Conference. (ERIC Document Reproduction Service
No. ED 432 305)
Barton, J., and Collins, A., (Eds). (1997) Portfolio
Assessment: A Handbook for Educators. Menlo Park, CA: Addison-Wesley.
Cambridge, B. L., Kahn, S., Tompkins, D. P., and Yancy,
K.B. (Eds.) (2001) Electronic portfolios - emerging practices in student,
faculty, and institutional learning. Washington, DC: American Association
for Higher Education
Elbow, P. & Belanoff, P. (1997). Reflections on an
explosion: Portfolios in the ‘90s and beyond. In: K. Yancy &
I. Weiser (Eds.), Situating portfolios: Four perspectives. Logan, Utah:
Utah State University Press
Graduate School of Education, George Mason University.
(1997) Ph. D. in Education Program: Portfolio Guidelines. http://gse.gmu.edu/programs/phd/portfolio_guidelines.html
Heiges, J. (1997). Portfolio for doctoral candidacy: A
veritable alternative. In: K. Yancy & I. Weiser (Eds.), Situating
portfolios: Four perspectives. Logan, Utah: Utah State University Press
Kimball, W., and Hanley, S. (1998). Anatomy of a portfolio
assessment system: Using multiple sources of evidence for credentialling
and professional development. In N. Lyons (Ed.), With portfolio in hand:
validating the new teacher professionalism. (pp. 189-201). New York:
Teachers College Press
Moss, P. (1998) Rethinking validity for the assessment
of teaching. In N. Lyons (Ed.), With portfolio in hand: validating the
new teacher professionalism. (pp. 202-219). New York: Teachers College
Press
Nitko, A., (2001). Educational assessment of students
(3rd ed.). Upper Saddle River, NJ: Prentice Hall.
Thomas, K. R., Lamson, S. L., & King, A. K. (2001).
Training teacher candidates to create web-based electronic professional
portfolios. Paper presentation at the annual meeting of the Association
of Teacher Educators, New Orleans, LA, February 17-21, 2001. (ERIC Document
Reproduction Service No. ED 458 191)
Yancy, K. B., & Weiser, I. (1997). Situating Portfolios:
Four perspectives. Logan, Utah: Utah State University Press
.
PDF Version