134 TOPIC 6 AUTHENTIC ASSESSMENT
Evidence of Authentic assessments offer The evidence is very
learnersÊ
proficiency or more direct evidence of indirect, particularly for
competency
application and construction claims of meaningful
Reliability and
validity of knowledge. application in complex,
For example, asking a student real-world situations.
to write a critique should For example, in MCQ, a
provide more direct evidence student effectively cannot
of that skill than asking the critique the arguments
student a series of multiple someone else has
choice, analytical questions presented (an important
about a passage. skill often required in the
real world).
Validity depends in part Validity is normally
upon whether the assessment determined by matching
simulates real-world tests of test items to the
ability. curriculum content.
It is difficult to ensure It is possible to have high
reliability because of the scoring reliability as the
subjective nature of scoring learnersÊ responses are
method (rubric) and the fixed. For example, there
presence of varied but is only one right answer
acceptable learnersÊ to a multiple-choice item.
responses.
Source: Adapted from Mueller (2005)
SELF-CHECK 6.1
1. What is authentic assessment?
2. State the other names used to describe authentic assessment.
3. Highlight three differences between authentic and traditional
assessments.
Copyright © Open University Malaysia (OUM)
TOPIC 6 AUTHENTIC ASSESSMENT 135
ACTIVITY 6.3
1. State the reasons why authentic assessment is a good replacement
for traditional assessment.
2. Give an example of authentic assessment.
Post your answers on the myINSPIRE online forum.
The strategy of asking students to perform real-world tasks that demonstrate
meaningful application of essential knowledge and skills is called authentic
assessment.
Authentic assessment is sometimes called performance assessment, alternative
assessment or direct assessment.
Authentic assessment has many advantages and traditional assessment
complements the authentic assessment.
An authentic assessment usually includes a task for students to perform and a
rubric by which their performance on the task will be evaluated.
Authentic assessment is a proven method of assessing human characteristics
that has been in use for decades.
Alternative assessment Indirect evidence
Backwards design Kinaesthetic
Contrived to real life Performance assessment
Direct assessment Student structured
Direct evidence
Copyright © Open University Malaysia (OUM)
136 TOPIC 6 AUTHENTIC ASSESSMENT
Brady, L. (2012). Assessment and reporting: Celebrating student achievement
(4th ed.). Melbourne, Australia: Pearson.
Kohn, A. (2006). The trouble with rubrics. English Journal, 95(4), 12ă15.
Linquist, E. F. (1951). Preliminary considerations in objective test construction. In
E.F. Linquist (Ed), Educational measurement (pp. 4ă22). Washington, DC:
American Council on Education.
Muller, J. (2005). The authentic assessment toolbox: Enhancing student learning
through online faculty development. Journal of Online Learning and
Teaching, 1(1), 1ă7.
Reeves, T. C., Herrington, J., & Oliver, R. (2002). Authentic activity as a model for
web-based learning. Presentation at the Annual Meeting of the American
Educational Research Association, New Orleans, FL.
Vander Ark, T. (2013). What is performance assessment? Retrieved from
http://gettingsmart.com/2013/12/performance-assessment/
Wiggins, G. (1990). The case for authentic assessment. Practical Assessment,
Research & Evaluation, 2(2), 1ă3.
Wiggins, G., & McTighe, J. (1998). Understanding by design. Alexandria, VA:
Association for Supervision and Curriculum Development (ASCD).
Copyright © Open University Malaysia (OUM)
Topic Project and
7 Portfolio
Assessments
LEARNING OUTCOMES
By the end of the topic, you should be able to:
1. Explain how to design an effective project for assessment;
2. Use different methods to assess group project work;
3. Discuss the usefulness of using projects as an assessment tool;
4. Describe the development of a portfolio; and
5. Discuss to what extent portfolios are useful as an assessment tool.
INTRODUCTION
Besides objective and essay tests, there are other methods of assessing students
that you can use. In this topic, we will focus on two other assessment methods,
project assessment and portfolio assessment. Both types of assessments are
examples of authentic assessment, which you have learnt in Topic 6. We will
discuss the project assessment first, followed by the portfolio assessment in this
topic. Since both project and portfolio assessments are examples of authentic
assessment, whatever we discuss under authentic assessment applies to them,
except that the points discussed in this topic are more specific.
Copyright © Open University Malaysia (OUM)
138 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
7.1 PROJECT ASSESSMENT
Most of us have done some form of project work in school or university and know
what a project is. However, when asked to define it, one will see varying
interpretations of the project and its purpose. „Projects‰ can represent a range of
tasks that can be done at home or in the classroom, by parents or groups of
students, quickly or over time. While project-based learning (PBL) also features
projects, in PBL, the focus is more on the process of learning and learner-peer-
content interaction than the end product itself.
A project is an activity in which time constraints have been largely removed and
it can be undertaken individually or by a group and usually involves a significant
element of work being done at home or out of school. Project work has its roots in
the constructivist approach which evolved from the work of psychologists and
educators such as Lev Vygotsky, Jerome Bruner, Jean Piaget and John Dewey.
Constructivism views learning as the result of mental construction wherein
students learn by constructing new ideas or concepts based on their current and
previous knowledge.
Most projects have the following common defining features (Katz & Chard, 1989)
which can also be considered as the strengths of using projects as an assessment
tool.
(a) It is a studentăcentred process. Students have the liberty to decide and plan
what to do and how to complete the project assigned though the selection of
a project may be determined by the teacher. If the choice is left to the
students, it probably requires the approval of the teacher. What is significant
is that students are involved in the beginning, middle and end of the project.
They will play an active role in the entire process and take ownership of their
project. They are actively involved as problem solvers, decision makers,
investigators, documenters and researchers. They should therefore find
projects fun, motivating and challenging.
(b) The content of a project and its work process are meaningful to students and
is directly observable in their environment. This is because projects normally
involve real-life problems and first-hand investigations. For instance, in
working on a project, students have to choose a knowledge area, delimit it
and formulate a problem or put forward questions. Then, they are required
to solve the problems and answer the questions through further work,
collection of materials and knowledge. Both the content gathered and the
work process involved are purposeful and reflective of real-life situations.
Project work allows for connections among school, life and work skills.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 139
(c) Projects can be planned with specific goals related to the curriculum.
Normally, work is planned in such a manner that it draws from knowledge
areas and skills in the current curriculum. It encourages students to break
away from the compartmentalisation of knowledge and instead involves
drawing upon different aspects of knowledge. It provides students with
opportunities to explore the interrelationships and interconnectedness of
topics within a subject and between subjects. For instance, the making of an
object requires handicraft skills, knowledge of materials, working methods
and uses of the object. Further, writing the project report requires language
skills. Technological supports will also enhance studentÊs learning. Thinking
skills are integral to project work. Project work thus involves drawing upon
different aspects of knowledge and skills from the curriculum and provides
students an integrated learning experience.
(d) The product/output of a project is tangible and visible, which can be shared
with the intended audience. It provides direct evidence of meaningful
application of knowledge and skills by the students. Teachers can directly
look at the project output/product to determine what they have learnt, while
parents need not have to grapple with statistical results to know the
performance of their children.
(f) Project work provides opportunities for reflective thinking and student self-
assessment. Students not only can reflect and self-assess what they have done
at the end, but also do so during the entire process, allowing continuous
learning to take place.
(g) Project work allows for multiple types of authentic assessment. For example,
in doing a project, students are required to use journals and diaries to
document the work process, portfolios to compile the project products,
reports to explain the work procedures, etc. All these outputs are useful
authentic evidence of studentsÊ performance that can be assessed.
(h) Project work provides an opportunity for students to explore different
approaches in solving problems. In project work, a teacher follows, discusses
and assesses the work in all its different phases. The teacher is the studentÊs
supervisor. When working on a project, the whole work process is as
important as the final result or product.
Copyright © Open University Malaysia (OUM)
140 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
Generally, there are two types of projects:
(a) Research-based Project
This is more theoretical in nature and may consist of posing a question,
formulating a problem or setting up some hypotheses. In order to answer the
question, solve the problem or confirm the assumptions, information must
be found, evaluated and used. This information can either be a result of their
own investigations or may be obtained from public sources without being a
pure reproduction. Such project work is usually presented as a research
report.
(b) Product-based Project
This can be the production of a concrete object, a service, a dance
performance, a film, an exhibition, a play, a computer programme and so
forth.
There are many types of effective projects. The following are some ideas for
projects:
(a) Survey of historical buildings in the studentÊs community;
(b) Study of the economic activities of people in the local community;
(c) Study of the transportation system in the district;
(d) Recreate a historical event;
(e) Develop a newsletter or website on a specific issue relevant to the school or
community (school safety, recycling, how businesses can save energy and
reduce waste);
(f) Compile oral histories of the local area by interviewing community elders;
(g) Produce a website as a „virtual tour‰ of the history of the community;
(h) Create a video of students graduating from a primary or secondary school;
(i) Create a wildlife or botanical guide for a local wildlife area;
(j) Create an exhibition on local products, local history or local personalities
using audiotapes, videotapes and photographs; and
(k) Investigate pollution of local rivers, lakes and ponds.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 141
The possibilities are endless. The key ingredient for any project idea is that it is
student-driven, challenging and meaningful. It is important to realise that project-
based instruction complements the structured curriculum. Project-based
instruction builds on and enhances what students learn through systematic
instruction. Teachers do not let students become the sole decision makers about
what project to do, nor do teachers sit back and wait for the students to figure out
how to go about the process, which may be very challenging (Bryson, 1994). This
is where the teacherÊs ability to facilitate and act as a coach, plays an important role
in the success of the project. The teacher will brainstorm ideas with the student to
come up with a number of project possibilities, discuss these possibilities and
options, help the students form a guiding question and be ready to help them
throughout the implementation process such as setting guidelines, due dates,
resource selection and so forth (Bryson, 1994).
SELF-CHECK 7.1
1. What is a project?
2. State the differences between a research-based project and a
product-based project.
ACTIVITY 7.1
Give examples of the two types of projects in your subject area or any
subject area.
Share your answer with your coursemates in the myINSPIRE online
forum.
7.1.1 What is Assessed Using Projects?
Project-oriented work is becoming increasingly common in working life. Project
competence, the ability to work together with others and take personal initiatives
and entrepreneurship are skills often required by employers. These competences
can be developed during project work which thus prepares students for working
life. Project work makes schooling more like the real world. In real life, we seldom
spend several hours listening to authorities who know more than we do and tell
us exactly what to do and how to do things. We ask questions of the person we are
Copyright © Open University Malaysia (OUM)
142 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
learning from. We try to link what the person is telling us with what we already
know. We bring our experiences and what we already know that is relevant to the
issue or task and say something about it.
You can see this with a class of young learners. When the teacher tells a story, little
kindergarten children raise their hands, eager to share their experiences with
something related to the story. They want to be able to apply their natural
tendencies to the learning process. This is how life is much of the time! By giving
project work, we open up areas in schooling where students can speak about what
they already know.
Project work is a learning experience which enables the development of certain
knowledge, skills and attitudes which prepares students for lifelong learning and
the challenges ahead (refer to Table 7.1).
Table 7.1: The Knowledge, Skills and Attitudes Achieved with Projects
Domains Learning Outcomes
Knowledge and
skills The ability to apply the knowledge and skills acquired in the project
application task.
Communication Examples:
Be able to choose a knowledge area and delimit a task or
problem.
Be able to choose relevant resources to complete the project.
Be able to draw up a project plan, implement it and if necessary
revise it.
Be able to apply creative and critical thinking skills in solving
problems.
The ability to communicate effectively, by presenting their ideas
clearly and coherently to specific audiences, in both the written and
oral forms.
Examples:
Be able to discuss with their supervising teacher how their work
is developing.
Be able to provide a written report of the project describing the
progress of work from the initial idea to final product.
Be able to produce a final product which is an independent
solution to the task or problem chosen.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 143
Collaboration The skills of working with others and in a team to achieve common
goals.
Independent Examples:
Learning
Be able to participate in group discussion actively.
Be able to listen actively the concerns of team members.
Be able to display a willingness to be a team player.
Be able to assess the strengths and weaknesses of partners.
Be able to recognise the contributions of team members.
Be able to play the roles assigned effectively and successfully.
The ability to learn on his/her own, self-reflect and take appropriate
actions to improve.
Examples:
Be able to document the progress of their work and regularly
report the process.
Be able to assess either in writing or verbally their work process
and results.
Be able to manage time for learning efficiently.
Source: Adapted from Harwell and Blank (1997)
SELF-CHECK 7.2
What are the knowledge, skills and attitudes evaluated using a project?
ACTIVITY 7.2
To what extent has project work been used as an assessment strategy in
Malaysian schools?
Discuss this matter with your coursemates in the myINSPIRE online
forum.
Copyright © Open University Malaysia (OUM)
144 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
7.1.2 Designing Effective Projects
There are many types of projects. There is no one correct way to design and
implement a project, but there are some questions and things to consider when
designing effective projects. You will be surprised that many teachers are not sure
why they use projects to assess their students. It is very important for everyone
involved to be clear about the learning goals of the project. Herman, Aschbacher
and Winters (1992) have identified five questions to consider when determining
learning goals:
(a) What important cognitive skills do I want my students to develop? (For
example, to use algebra to solve everyday problems, to write persuasively);
(b) What social and affective skills do I want my students to develop? (For
example, develop teamwork skills);
(c) What metacognitive skills do I want my students to develop? (For example,
reflect on the research process they use, evaluate its effectiveness and
determine methods of improvement);
(d) What types of problems do I want my students to be able to solve? (For
example, know how to do research, apply a scientific method); and
(e) What concepts and principles do I want my students to be able to apply? (For
example, apply basic principles of biology and geography in their lives,
understand cause-and-effect relationships).
In designing project work for assessment, the teacher should also develop an
outline that explains the projectÊs essential elements and his or her expectations for
each project. Although the outline can take various forms, it should contain the
following elements (Bottoms & Webb, 1998):
(a) Situation or Problem
A sentence or two describing the issue or problem that the project is trying
to address. For example, the pollution levels in rivers, transportation
problems in urban centres, the price of essential items are increasing, crime
rate in squatter areas, youths loitering in shopping complexes, students in
Internet cafes during school hours and so forth.
(b) Project Description and Purpose
A concise explanation of the projectÊs ultimate purpose and how it addresses
the situation or problem. For example, students will research, conduct
surveys and make recommendations on how students can help reduce
pollution of rivers. Results will be presented in a newsletter, information
brochure, exhibition or website.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 145
(c) Performance Specifications
A list of criteria or quality standards the project must meet.
(d) Rules
Guidelines for carrying out the project include timeline and short-term goals,
such as to have interviews and research completed by a certain date.
(e) List of Project Participants with Roles Assigned
Roles of team members and if members of the community are involved,
identify their roles.
(f) Assessment
How the studentÊs performance will be evaluated. In project work, the
learning process is being evaluated as well as the final product.
What concepts and principles do I want my students to be able to apply? (For
example, apply basic principles of biology and geography in their lives,
understand cause-and-effect relationships).
Steinberg (1998) provides a checklist, which is called the Six AÊs Project Checklist,
for the design of effective projects (refer to Table 7.2). The checklist can be used
throughout the process to help both teacher and student plan and develop a
project, as well as to assess whether the project is successful in meeting
instructional goals.
Table 7.2: The Six AÊs Project Checklist
Six AÊs Project Questions Checklist
Authenticity
Does the project stem from a problem or question that is meaningful
Academic to the student?
rigour
Is the project similar to one undertaken by an adult in the
community or workplace?
Does the project give the student the opportunity to produce
something that has value or meaning to the student beyond the
school setting?
Does the project enable the student to acquire and apply knowledge
central to one or more discipline areas?
Does the project challenge the student to use methods of inquiry
from one or more disciplines (such as to think like a scientist)?
Does the student develop higher-order thinking skills (such as
searching for evidence, using different perspectives)?
Copyright © Open University Malaysia (OUM)
146 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
Applied Does the student solve a problem that is grounded in real life and/or
learning work (such as design a project, organise an event)?
Active Does the student need to acquire and use skills expected in high-
exploration performance work environments (such as teamwork, problem-
solving, communication or technology)?
Adult
relationships Does the project require the student to develop organisational and
self-management skills?
Assessment
practices Does the student spend significant amount of time doing work in
the field, outside school?
Does the project require the student to engage in real investigative
work, using a variety of methods, media and sources?
Is the student expected to explain what he/she learnt through a
presentation or performance?
Does the student meet and observe adults with relevant experience
and expertise?
Is the student able to work closely with at least one adult?
Do adults and the students collaborate on the design and assessment
of the project?
Does the student reflect regularly on his or her learning, using clear
project criteria that he or she has helped to set?
Do adults from outside the community help the student develop a
sense of the real-world standards from this type of work?
Is the studentÊs work regularly assessed through a variety of
methods, including portfolios and exhibitions?
Source: Steinberg (1998)
In implementing the project, it is also important to ensure that the following
questions are addressed:
(a) Do the students have easy access to the resources they need? This is
especially important if a student is using specific technology or subject-
matter expertise from the community;
(b) Do the students know how to use the resources? Students who have minimal
experience with the computer, for example, may need extra assistance in
utilising it;
(c) Do the students have mentors or coaches to support them in their work? This
can be in-school or out-of-school mentors; and
(d) Are students clear on the roles and responsibilities of each person in the
group?
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 147
ACTIVITY 7.3
1. What are some of the factors you should consider when designing
project work for students in your subject area?
2. Give examples of projects you have included or can include in the
teaching and evaluation of your subject area.
Post your answers on the myINSPIRE online forum.
7.1.3 Possible Problems with Project Work
Teachers intending to use projects both as an instructional and assessment tool
should be aware of certain problem areas. Be as specific as possible in determining
outcomes so that both the student and the teacher understand exactly what is to
be learnt. In addition, be aware of the following problems when undertaking
project-based instruction:
(a) Aligning project goals with curriculum goals can be difficult and requires
careful planning. For example, if one of the curriculum goals is to develop
the reading skills of learners, the project planned should enable learners to
acquire the skills in the process. Very often, the skills are assessed but the
scope for learning is not made available.
(b) Parents tend to be exam-oriented. To the parents, the best way to assess
learning is through tests and examinations. They cannot see how doing a
project is related to the overall assessment of learning. They are thus not
supportive of projects and consider doing projects a waste of student
learning time and resources.
(c) Students are not clear as to what is required of them. This happens when the
assigned projects do not have adequate and clear structure and guidelines.
Students are also not given proper guidance on how to carry out the projects.
(d) Projects often take longer time to complete and teachers need a lot of time to
prepare good authentic projects, to manage and monitor their
implementation.
Copyright © Open University Malaysia (OUM)
148 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
(e) Teachers are not traditionally prepared to integrate curriculum content into
real-world activities. Besides, they may not be familiar with how they should
assess projects. There is thus a need for intensive staff development to
prepare them for the job.
(f) Resources needed for project work are not confined to the usual classroom
materials such as paper and pencil. For instance, a project to develop a
website requires a computer, special Internet program and other Internet
facilities which may not be readily available to the learners. Such resources
also involve cost. Support from school administration is needed.
(g) Scoring studentsÊ project work can be a daunting task. Project work normally
involves the assessment of diverse competencies of the students. It also
allows for the production of many diverse outputs by the students. Assessing
project work is time-consuming. Besides, teachers need to be specially
trained to carry out this type of assessment especially when the project is
undertaken by a group of student. Fairness in assessment among group
members is an issue that needs special attention.
7.1.4 Group Work in Projects
A group project requires two or more students to work together on a longer
project. Working in groups has become an accepted part of learning as a
consequence of widely recognised benefits of collaborative group work for student
learning. When groups work well, students learn more and produce higher quality
learning outcomes. What are some benefits of group work in projects?
(a) Group Work Can Enhance the Overall Quality of Student Learning
Groups that work well together can achieve much more than an individual
working on his or her own. A broader range of skills can be applied to
practical activities and sharing and discussing ideas can play a pivotal role
in deepening an individual studentÊs understanding of a particular subject
area. This is because working in a group enables him to examine topics from
the perspectives of others. When an individual is required to discuss a topic
and negotiate how to address it, he is forced to listen to other peopleÊs ideas.
Their ideas will then influence his own thinking and broaden his horizons.
His group members are not just fellow learners, they are also his teachers.
Besides, being part of a team will help him develop his interpersonal skills
such as speaking and listening. Group work will also help him find his own
strengths and weaknesses (for example, he may be a better leader than
listener, or he might be good at coming up with the „big ideas‰ but not so
good at putting them into action). Enhanced self-awareness will help his
approach to learning.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 149
(b) Group Work Can Improve Peer Learning
Group work enhances peer learning. Students learn from each other and
benefit from activities that require them to articulate and test their
knowledge. Group work provides an opportunity for students to clarify and
refine their understanding of concepts through discussion and rehearsal
with peers. Many, but not all students recognise the value of group work to
their personal development and of being assessed as a member of a group.
Working with a group and for the benefit of the group also motivates some
students. Group assessment helps some students develop a sense of
responsibility. A student working in a group on a project may think, „I felt
that because one is working in a group, it is not possible to slack off or to put
things off. I have to keep working, otherwise I would be letting other people
down.‰
(c) Group Work Can Help Develop Generic Skills Sought by Employers
As a direct response to the objective of preparing graduates with the capacity
to function successfully as team members in the workplace, there has been a
trend in recent years to incorporate generic skills alongside traditional
subject-specific knowledge in the expected learning outcomes in higher
education. Group work can facilitate the development of skills, which
include:
(i) Teamwork skills (skills in working within team dynamics and
leadership skills);
(ii) Analytical and cognitive skills (analysing task requirements,
questioning, critically interpreting material and evaluating the work of
others);
(iii) Collaborative skills (conflict management and resolution, accepting
intellectual criticism, flexibility and negotiation and compromise); and
(iv) Organisational and time management skills. A student might say,
„Having to do group work has changed the way I worked. I could not
do it all the night before. I had to be more organised and efficient.‰
(d) Group Work May Reduce the Work Load Involved in Assessing, Grading
and Providing Feedback to Students
Group work and group assessment in particular, is sometimes implemented
in the hope of streamlining assessment and grading tasks. In simple terms, if
students submit group assignments, then the number of pieces of work to be
assessed can be vastly reduced. This prospect might be particularly attractive
for staff teaching large classes.
Copyright © Open University Malaysia (OUM)
150 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
SELF-CHECK 7.3
1. What are some problems in the implementation of project work
and how would you solve them?
2. What are the benefits of group work in projects?
7.1.5 Assessing Project Work
Assessing student performance on project work is quite different from an
examination using objective tests and essay questions. It is possible that students
might be working on different projects; some may be working in groups while
others are working alone. This makes the task of assessing student progress even
more complex compared with a paper-and-pencil test where everyone is evaluated
using one marking scheme. Table 7.3 illustrates the general marking scheme for
projects.
Marks Table 7.3: General Marking Scheme for Projects
100ă90%
Criteria
89ă80%
Exceptional and distinguished work of a professional standard.
79ă70% Outstanding technical and expressive skills.
Work demonstrating exceptional creativity and imagination.
69ă60% Work displaying great flair and originality.
Excellent and highly developed work of a professional standard.
Extremely good technical and expressive skills.
Work demonstrating a high level of creativity and imagination.
Work displaying flair and originality.
Very good work which approaches professional standard.
Very good technical and expressive skills.
Work demonstrating good creativity and imagination.
Work displaying originality.
A good standard of work.
Good technical and expressive skills.
Work displaying creativity and imagination.
Work displaying some originality.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 151
59ă50% A reasonable standard of work.
49ă40% Adequate technical and expressive skills.
39ă30% Work displaying competence in the criteria assessed, but which may
29ă20%
19ă10% be lacking some creativity or originality.
9ă1%
A limited, but adequate standard of work.
Limited technical and expressive skills.
Work displaying some weaknesses in the criteria assessed and
lacking creativity or originality.
Limited work which fails to meet the required standard.
Weak technical and expressive skills.
Work displaying significant weaknesses in the criteria assessed.
Poor work. Unsatisfactory technical or expressive skills.
Work displaying significant or fundamental weaknesses in the
criteria assessed.
Very poor work or work where very little attempt has been made.
A lack of technical or expressive skills.
Work displaying fundamental weaknesses in the criteria assessed.
Extremely poor work or work where no serious attempt has been
made.
Source: Chard (1992)
Product, Process or Both?
According to Bonthron and Gordon (1999), from the onset, you should be clear
about the following:
(a) Whether you are going to assess the product of the group work or both
product and process.
(b) If you intend to assess process, what proportions of the marks are you going
to allocate for process and based on what criteria? And how are you going to
use the criteria to assess process?
(c) What criteria are you planning to use to assess the product and how will the
marks be distributed?
Some educators believe there is a need to assess the processes within groups as
well as the products or outcomes. What exactly does process mean? Both teachers
and students must be clear what process means. For example, if you want to assess
„the level of interaction‰ among students in the group, they should know what
„high‰ or „low‰ interaction means. Should the teacher be involved in the working
Copyright © Open University Malaysia (OUM)
152 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
of each group or should it rely on self or peer assessment? Obviously, being
involved in so many groups would be physically impossible for the teacher. So,
how do you measure „process‰? Some educators may say, „I donÊt care what they
do in their groups. All IÊm interested in is the final product and how they arrive at
their results is their business.‰
However, to provide a more balanced assessment, there is growing interest in both
the process and product of group work and the issue that arises is, „What
proportion of assessment should focus on product and what proportion should
focus on process?‰
The criteria for the evaluation of group work can be determined by teachers alone
or by both teachers and students through consultation. Group members can be
consulted on what should be assessed in a project through consultation with the
teacher. Obviously, you have to be clear about the intended learning outcomes of
the project in your subject area. It is a useful starting point for determining criteria
for assessment of the project. Once these broader learning outcomes are
understood, you can establish the criteria for marking the project. Generally, it is
easier to establish criteria for measuring the product of project work and much
more difficult to measure the processes involved in project work. However, it is
suggested that evaluation of product and process can be done separately rather
than attempting to do both at once.
Who Gets the Marks ă Individuals or the Group?
Most projects involve more than one student and the benefits of group work have
been discussed earlier. A major problem of evaluating projects involving group
work is how to allocate marks fairly among group members. As exclaimed by a
student, „I would like my teacher to tell me what amount of work and effort will
get what mark.‰ Other concerns would be, „Do all students get the same mark
even though not all students put in the same effort?‰ and „Are marks given for the
individual contribution of team members?‰
These are questions that bother teachers, especially when it is common to find free
loaders or sleeping partners in group projects. The following are some suggestions
how group work may be assessed:
(a) Shared Group Mark
All group members receive the same mark for the work submitted regardless
of individual contribution. It is a straightforward method that encourages
group work where group members sink or swim together. However, it may
be perceived as unfair by better students who may complain that they are
unfairly disadvantaged by weaker students and the likelihood of „sleeping
partners‰ is very high.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 153
(b) Share-out Marks
The students in the group decide how the total number of marks should be
shared between them. For example, a score of 40 is given by the teacher for
the project submitted. There are five members in the group and so the total
score possible is 5 40 = 200. The students then share the 200 marks based
on the contribution of each of the five students; which may be 35, 45, 42, 38
and 40. This is an effective method if group members are fair, honest and do
not have ill feelings towards each other. However, there is the likelihood for
the marks to be equally distributed to avoid ill feelings among group
members.
(c) Individual Mark
Each student in the group submits an individual report based on the task
allocated or on the whole project.
(i) Allocated Task
From the beginning, the project is divided into different parts or tasks
and each student in the group completes his or her allocated task that
contributes to the final group product and gets the marks for that task.
This method is a relatively objective way of ensuring individual
participation and may motivate students to work hard on their task or
part. The problem is breaking up the project into tasks that are exactly
equal in size or complexity. Also, the method may not encourage group
collaboration and some members may slow down progress.
(ii) Individual Report
Each student writes and submits an individual report based on the
whole project. The method ensures individual effort and may be
perceived as fair by students. However, it is difficult to determine how
the individual reports should differ and students may unintentionally
commit plagiarism.
(d) Individual Mark (Examination)
Use examination questions that specifically target the group projects, and can
only be answered by students who have been thoroughly involved in the
project. This method may motivate students to learn from the group project
including learning from the other members of the group. However, it may
not be effective because students may be able to answer the questions by
reading the group reports. In the Malaysian context, a national examination
may not be able to include such questions as it involves hundreds of
thousands of students.
Copyright © Open University Malaysia (OUM)
154 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
(e) Combination of Group Average and Individual Marks
The group mark is awarded to each member with a mechanism for adjusting
for individual contributions. This method may be perceived to be fairer than
shared group mark. However, it means additional work for teachers trying
to establish individual contribution.
ACTIVITY 7.4
Which of the five methods of assessing group work would you use in
evaluating project work in your subject area? Give reasons for your
choice.
Post your answer on the myINSPIRE online forum.
7.1.6 Evaluating Process in a Project
The assessment of a group product is rarely the only assessment taking place in
group activities. The process of group work is increasingly recognised as an
important element in the assessment of group work. Moreover, where group work
is marked solely on the basis of product and not process, there can be differences
in individual grading that are unfair and unacceptable. The following are the
elements which are considered in evaluating process:
(a) Peer/Self Evaluation of Roles
Students rate themselves as well as other group members on specific criteria,
such as responsibility, contributing ideas and finishing tasks. This can be
done through various grading forms (refer to Figure 7.1) or having students
write a brief essay on the group membersÊ strengths and weaknesses.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 155
Figure 7.1: Checklist for evaluating processes involved in project work
Source: Sutherland (2003)
Copyright © Open University Malaysia (OUM)
156 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
(b) Individual Journals
Students keep a journal of events that occur in each group meeting. These
include who attended, what was discussed and plans for future meetings.
These can be collected and periodically read by the instructor, who
comments on progress. The instructor can provide guidance for the group
without directing them.
(c) Minutes of Group Meetings
Similar to journals are minutes for each group meeting, which are periodically
read by the instructor. These include who attended, tasks completed, tasks
planned and contributors to various tasks. This provides the instructor with
a way of monitoring individual contributions to the group.
(d) Group and Individual Contribution Grades
Instructors can divide the project grade into percentage of individual and
group contribution. This is especially beneficial if peer and self-evaluations
are used.
Logs can potentially provide plenty of information to form the basis of
assessment while keeping minutes helps members to focus on the process
which is a learning experience in itself. These techniques may be perceived
as a fair way to deal with „shirkers‰ and outstanding contributions.
However, reviewing logs can be time consuming for teachers and students
may need a lot of training and experience in keeping records. Also, emphasis
on second-hand evidence may not be reliable.
7.1.7 Self-assessment in Project Work
Self-assessment is a process by which students learn about themselves; for
example, what they have learnt about the project, how they have learnt and how
they have reacted in certain situations when carrying out the project. Involving
students in the assessment process is an essential part of balanced assessment.
When students become partners in the learning process, they gain a better sense of
themselves as readers, writers and thinkers. Some teachers maybe uncomfortable
with self-assessment because traditionally teachers are responsible for all forms of
assessment in the classroom and here we are asking students to assess themselves.
Self-assessment can take many forms such as:
(a) Discussion involving the whole class or small groups;
(b) Reflection logs;
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 157
(c) Self-assessment checklist or inventories; and
(d) Teacher-student interviews.
These types of self-assessment share a common theme; they ask students to review
their work to determine what they have learnt and if areas of confusion still exist.
Although each method may differ slightly, they all include enough time for
students to consider thoughtfully and evaluate their own progress.
Since project learning is student-driven, assessment should be student-driven as
well. Students can keep journals and logs to continually assess their progress. A
final reflective essay or log can allow students and teachers to understand thinking
processes, reasoning behind decisions, ability to arrive at conclusions and
communicate what they have learnt. According to Edwards (2000), the following
are some questions a student can ask himself or herself while self-assessing:
(a) What were the projectÊs successes?
(b) What might I do to improve the project?
(c) How well did I meet my learning goals? What was most difficult about
meeting the goals?
(d) What surprised me most about working on the project?
(e) What was my groupÊs best team effort? Worst team effort?
(f) How do I think other people involved with the project felt it went?
(g) What were the skills I used during this project? How can I practise these skills
in the future?
SELF-CHECK 7.4
1. Explain how process can be measured in a group project work.
2. List some of the problems with the evaluation of process.
ACTIVITY 7.5
Do you think process should be assessed?
Justify your answer in the myINSPIRE online forum.
Copyright © Open University Malaysia (OUM)
158 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
7.2 WHAT IS A PORTFOLIO?
A portfolio is a collection of pieces of student work presented for assessment.
However, it does not contain all the work a student does. It may contain examples
of „best‰ works or examples from each of several categories of work, for example,
a book review, a letter to a friend, a creative short story and a persuasive essay. A
studentÊs portfolio may have one or several goals. The student will select and
submit works to meet these goals. The works submitted should provide evidence
of studentsÊ progress towards the goals and reflect both student production and
process.
A portfolio is therefore not a pile of student work that accumulates over a semester
or year. Rather, a portfolio contains a purposefully selected subset of student work
which reflects his or her efforts, progress and achievements in different areas of
the curriculum. Some people may associate portfolios with the stock market where
a person or organisation keeps a portfolio of stocks and shares owned. A portfolio
can be defined as a container that holds evidence of an individualÊs skills, ideas,
interests and accomplishments. The organised collection of contents such as text,
files, photos, videos and more to tell that story, are generically referred to as
artefacts and evidence of what students have learnt. These artefacts are usually
accompanied by studentsÊ reflection.
The particular purposes of portfolio determine the number and type of items to be
included, the process for selecting the items, how and whether students respond
to the items selected.
Portfolios offer a way of assessing student learning that is different from
traditional methods. Portfolio assessment provides the teacher and students an
opportunity to observe students in a broader context: taking risks, developing
creative solutions and learning to make judgements about their own
performances.
(Paulson, Paulson & Meyer, 1991)
Portfolios typically are created for one of three purposes; to show growth, to
showcase current abilities and to evaluate cumulative achievement. Many
educators who work with portfolios consider the reflection component the most
critical element of a good portfolio. Simply selecting samples of work can produce
meaningful stories about students and others can benefit from „reading‰ these
stories.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 159
The students themselves are missing significant benefits of the portfolio process if
they are not asked to reflect upon the quality and growth of their work. As Paulson
et al. (1991) stated, „The portfolio is something that is done by the student, not to
the student.‰ Most importantly, it is something done for the student. The student
needs to be directly involved in each phase of the portfolio development to learn
the most from it and the reflection phase holds the most promise for promoting
studentÊs growth.
Portfolios are sometimes described as portraits of a personÊs accomplishments.
Using this metaphor, we can consider a student portfolio a self-portrait, but one
that has benefited from guidance and feedback from a teacher and sometimes from
other students.
7.2.1 What is Portfolio Assessment?
Increasingly, portfolio assessment is gaining acceptance as an assessment strategy
seeking to present a more holistic view of the learner. The collection of works by
students are assessed and hence the term portfolio assessment. However, some
suggest that portfolios are not really assessments at all because they are just
collections of previously completed assessments. In portfolio assignment, students
are in fact performing authentic tasks which capture meaningful application of
knowledge and skills. Their portfolios often tell compelling stories of the growth
of the studentsÊ talents and showcase their skills through a collection of authentic
performances.
The portfolio provides for continuous and ongoing assessment (i.e. formative
assessment) as well as assessment at the end of a semester or a year (i.e. summative
assessment). Emphasis is more on monitoring studentsÊ progress towards
achieving the learning outcomes of a particular subject, course or programme.
Portfolio assessment has been described as multidimensional because it allows
students to include different aspects of their works such as essays, project reports,
performance on objective tests, objects or artefacts they have produced, poems,
laboratory reports and so forth. In other words, the portfolio contains samples of
work over an entire semester, term or year, rather than single points in time, such
as during examination week only.
Using portfolios introduces students to an evaluation format which they may need
to become familiar as more schools adopt portfolio assessment. Although many
portfolios reflect long-term projects completed over a period of time, they do not
have to be that way. Teachers can have students create portfolios of their work for
a particular unit. That portfolio might count as a project for that particular topic of
study. Though portfolios assessment is currently quite popular in our school
system, there are still teachers who are uncomfortable to use it as an assessment
Copyright © Open University Malaysia (OUM)
160 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
tool. These teachers may have the thinking that the portfolio is a very subjective
form of assessment. They may be unsure of the purpose of a portfolio and its uses
in the classroom. To them, there is also the question of how the portfolio can be
most effectively used to assess student learning. The situation can be overcome if
these teachers understand the purpose of portfolios, how the portfolios can be used
to evaluate their studentsÊ work and how grades will be determined.
Portfolio assessment represents a significant shift in thinking about the role of
assessment in education. Teachers who use this strategy in the classroom have
shifted their philosophy of assessment from merely comparing achievement
(based on grades, test score, percentile rankings) towards improving studentsÊ
achievement through feedback and self-reflection. Teachers should convey to
students the purpose of the portfolio, what constitute quality work and how the
portfolio is graded.
7.2.2 Types of Portfolios
There are two main types of portfolios, namely process-oriented and product-
oriented portfolios (refer to Table 7.4).
Table 7.4: Types of Portfolios
Process-oriented Portfolios Product-oriented Portfolios
These portfolios tell a story about the These portfolios contain the works of a
student and how the learner has grown. It student which he or she considers the best.
will include earlier drafts and how these The aim is to document and reflect on the
drafts have been improved upon. For quality of the final products rather than the
example, the first draft of a poem written process that produced them. The student is
by a Year Three student is reworked based required to collect all his or her work at the
on the comments by the teacher and the end of the semester, at which time he or
student reflecting on his or her work. All she must select those works which are of
the drafts and changes made are kept in the highest quality. Students could be left
the portfolio. In this manner, student to make the decision themselves or the
works can be compared by providing teacher can set the criteria on what a
evidence of how the studentÊs skills have portfolio must contain and the quality of
improved. the works to be included.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 161
SELF-CHECK 7.5
1. What is portfolio assessment?
2. Describe two main types of portfolios.
7.2.3 Developing a Portfolio
According to Epstein (2006), the design and development of a portfolio involves
four main steps as follows:
(a) Collection
This step simply requires students to collect and store all of their work.
Students have to get used to the idea of documenting and saving their work
which they may not have done before. Questions involved in this step are:
(i) How should the work be organised? By subject or by themes?
(ii) How should the work be recorded and stored?
(iii) How to get students to form the habit of documenting evidence?
(b) Selection
This will depend on whether it is a process or product portfolio and the
criteria set by the teacher. Students will go through the work collected and
select certain works for their portfolio. This might include: examination
papers and quizzes, audio and video recordings, project reports, journals,
computer work, essays, poems, artwork and so forth. Questions related to
this step are:
(i) How does one select? What is the basis of selection?
(ii) Who should be involved in the selection process?
(iii) What are the consequences of not completing the portfolio?
(c) Reflection
This is the most important step in the portfolio process. It is reflection that
differentiates the portfolio from a mere collection of student work. Reflection
is often done in writing but it can also be done orally. Students are asked why
they have chosen a particular product or work (such as essay); and how it
Copyright © Open University Malaysia (OUM)
162 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
compares with other work, what particular skills and knowledge were used
to produce it (such as the essay) and how it can be further improved.
Questions related to this step are:
(i) Should students reflect on how or why they chose certain works?
(ii) How should students go about the reflection process?
(d) Connection
As a result of „reflection‰, students begin to ask themselves, „Why are we
doing this?‰ They are encouraged to make connections between their
schoolwork and the value of what they are learning. They are also
encouraged to make connections between the work included in their
portfolio with the world outside the classroom. They learn to exhibit what
they have done in school to the community. Questions to ask are:
(i) How is the cumulative effect of the portfolio evaluated?
(ii) Should students exhibit their works?
7.2.4 Advantages of Portfolio Assessment
Portfolio assessments have been gaining importance as an assessment strategy in
educational institutions because of their benefits to teaching and learning.
However, like other assessment methods, the benefits of portfolio assessments also
bring along some problems. Let us first look at the advantages of using portfolios
as an assessment tool.
(a) Allows Assessment of Creativity and Higher-level Cognitive Skills
It has frequently been suggested that paper-and-pencil tests (objective and
essay tests) are not able to assess all the learning outcomes in a particular
subject area. For example, many higher-level cognitive skills and the affective
domain (feelings, emotions, attitudes and values) are not adequately
assessed using traditional assessment methods. However, portfolio
assessments allow for the assessment of studentsÊ higher-level cognitive
skills such as critical and creative thinking skills. For instance, they can be
assessed on how critical they are in individual reflection. Likewise, they can
be assessed how creative they are in the selection and presentation of their
works for portfolio assessments. Materials compiled by students in the
portfolio development process also provide evidence about his or her growth
in the affective domain such as self-confidence, diligence, attention to detail
and positive attitude towards learning.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 163
(b) Continous, Ongoing Process
Portfolio assessments are an ongoing process. Hence, they not only provide
an opportunity for the teacher to trace or monitor change and growth over a
period of time, but also provide an opportunity for students to reflect their
own learning and thinking. Teachers have an opportunity to monitor their
understanding and approaches to solving problems and decision making
(Paulson et al., 1991), while upon reflection, students can identify where they
have gone wrong or how they can improve themselves. Emphasis in portfolio
assessment is on improving studentÊs achievement rather than ranking
students according to their performance on tests.
Portfolio assessments are both formative and summative. Since the
assessment is a continuous and ongoing process, it allows students to reflect
on their own learning and thinking and allows teachers to monitor studentsÊ
progress and to provide feedback. Through teacher feedback and self-
reflection, students improve their achievement. The assessment of this
learning process is formative while the assessment of the documents that are
being compiled as well as assessment of the final products at the end of the
semester/year are summative.
(c) Multidimensional
Portfolio assessments are multidimensional in that they allow for inclusion
of different aspects of studentsÊ works and also samples of work over time,
i.e. over a semester/year. Portfolios are thus a rich source of evidence on
student learning and development, providing a more complete and holistic
picture of studentsÊ achievement.
(d) Encourages Self-assessment
Portfolio assessments involve students in the assessment process. Students
self-assess their own work and decide what to include as the evidence of their
growth and performance. They judge their work using explicit criteria to
identify strengths and weaknesses and monitor their own progress. By self-
evaluating their own work, students will become more accountable and be
more responsible of their own learning. Student learning has also become
more meaningful. For self-evaluation to take place, teachers should
constantly invite students to reflect their growth and performance as
students. The teachers should convey to students the purpose of the
portfolio, what constitutes quality work and how the portfolio is graded.
Feedback enables learners to reflect on what they are learning and why.
Copyright © Open University Malaysia (OUM)
164 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
(e) Tangible Outcomes
In portfolio assessments, the portfolios give parents and teachers concrete
examples of studentsÊ development over time as well as their current skills
and abilities. The assessment outcomes are also tangible and more
meaningful than the numeric statistical results. For example, through the
compositions written and compiled by a student in the portfolio, parents and
teachers cannot only see how he or she has mastered the writing skills but
also how he or she has improved over time.
(f) Individualised
Portfolio assessments are individualised, meaning that studentsÊ portfolios
will be assessed separately. This allows teachers to see their students as
individuals, each with his or her own unique characteristics, needs and
strengths. With this understanding, teachers can adapt their instruction to
the learning needs and styles of the students.
7.2.5 Disadvantages of Portfolio Assessment
There are also disadvantages in portfolio assessment.
(a) Time Consuming
Extra time is needed to plan an assessment system as the assessment involves
multiple learning outcomes. The problem is further compounded if a large
group of students need to be assessed. The portfolio outputs can be varied
and require expert judgement. Thus, assessing portfolios is time consuming
for teachers and the data from portfolios will be difficult to analyse.
(b) Need for Constant Feedback and Mentoring
Moreover, for portfolio assessments to be beneficial to students, teachers
need to provide constructive feedback on the work included in the studentsÊ
portfolios and on the portfolios as a whole. Teachers also need to provide
guidance through portfolio conferences about how best to construct a
portfolio for a specific purpose. Scheduling individual portfolio conferences
is difficult and the length of each conference may interfere with other
instructional activities. There is thus a need to ensure that the benefits of
portfolio assessments justify the investment of time by the teachers.
(c) Poor Reliability
Scoring portfolios involves extensive use of subjective evaluation procedures
and thus open to the question of reliability. Assessment is useless if data
obtained is unreliable. According to Nitko (2001), the reliability of portfolio
results is typically in the 0.4 to 0.6 range. This indicates that as much as
60 per cent of the variability in portfolio scores is the results of measurement
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 165
error. This should give all educators a reason to be cautious when using the
results of portfolio assessments in assigning course grades, certifying
achievement or making high-stake decisions. Part of the poor reliability
comes from the difficulty of establishing clear scoring criteria for the large
and diverse sets of materials that are included. Poor reliability is also due to
a lack of standardisation that leads to incomparability of portfolio entries that
different students choose to include.
(d) Energy, Skills and Resources
Society is still strongly oriented towards grades and test scores and in
addition, most universities and colleges still use test scores and grades as the
main admission criteria. Parents who are exam-oriented cannot see how
keeping portfolios is related to the overall assessment of learning. They are
thus usually not supportive of portfolio assessments and consider them a
waste of student learning time and resources. To them, the best way to assess
learning is through tests and examinations.
An important aspect of portfolio assessments is reflection. In fact, this is the
most important step in the portfolio process. As mentioned earlier in
subtopic 7.2.3, it is reflection that differentiates the portfolio from a mere
collection of student work. Reflection is often done in writing but it can also
be done orally. They will be asked to reflect on what they are learning and
why. To be able to do this, students must possess good metacognitive skills
to think about what they are thinking. This can be a problem to students,
especially those who are underachievers.
In summary, portfolio assessments have significant strengths and weaknesses. On
the positive side, they provide a broad framework for examining a studentÊs
progress, encourage student participation in the assessment process and
strengthen the relationship between instruction and assessment. On the down
side, they demand considerable time, energy and a certain degree of expertise on
the part of teachers as well as the students, and have questionable reliability.
7.2.6 How and When Should Portfolios be Assessed?
If the purpose of the assessment is to demonstrate progress, the teacher could make
judgements about the evidence of progress and provide those judgements as
feedback to the student. The student could self-assess progress to check whether
the goals have been met or not.
Copyright © Open University Malaysia (OUM)
166 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
The portfolio is more than just a collection of student work. The teacher may assess
and assign grades to the process of assembling and reflecting upon the portfolio of
a studentÊs work. The students might have also included reflections on growth, on
strengths and weaknesses, on goals that were or are to be set, on why certain
samples tell a certain story about them or on why the contents reflect sufficient
progress to indicate completion of designated standards. Some of the process skills
may also be part of the teacherÊs or schoolÊs or districtÊs standards. So, the portfolio
provides some evidence of attainment of those standards. Any or all of these
elements can be evaluated and/or graded.
The portfolio assignments can be assessed or graded with a rubric. Rubric is useful
in avoiding personal judgement in assessing a complex product such as a portfolio.
Clear criteria for assessment, including what must be included in the portfolio and
rubrics are vital to a successful portfolio assessment. Rubric can provide some
clarity and consistency in assessing and judging the quality of the content and the
elements making up that content. Moreover, application of a rubric increases the
likelihood of consistency among the teachers who are assessing the portfolios.
Table 7.5 is a sample portfolio rubric that may be used for self-assessment and peer
feedback.
Table 7.5: Portfolio Rubric
Criteria Unsatisfactory Emerging Proficient Exemplary Rating
Selection of
artefacts The artefacts Some of the Most artefacts All artefacts
and work artefacts and and work and work
Descriptive samples do work samples are samples are
text not relate to samples are related to the clearly and
the purpose of related to the purpose of directly
the portfolio. purpose of the portfolio. related to the
the portfolio. purpose of the
No artefacts Most of the portfolio. A
are Some of the artefacts are wide variety
accompanied artefacts are accompanied of artefacts is
by a caption accompanied by a caption included.
that clearly by a caption that clearly
explains the that clearly explains the All artefacts
importance of explains the importance of are
the item importance the item work accompanied
including title, of the item including by a caption
author and including title, author that clearly
date. title, author and date. explains the
and date. importance of
the item
including title,
author and
date.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 167
Reflection The A few of the Most of the All reflections
Citations reflections reflections reflections clearly explain
do not explain explain how the artefacts
explain growth and growth and demonstrate
growth or include goals include studentsÊ growth,
include for continued goals for competencies,
goals for learning. continued accomplishments
continued learning. and include goals
learning. for continued
learning.
The A few Most of the All reflections
reflections reflections reflections illustrate the
do not illustrate the illustrate the ability to
illustrate the ability to ability to effectively
ability to effectively effectively critique work and
effectively critique work critique provide
critique and provide work and suggestions for
work or suggestions provide constructive
provide for suggestions practical
suggestions constructive for alternatives.
for practical constructive
constructive alternatives. practical
practical alternatives.
alternatives.
No images, Some of the Most All images,
media or images, images, media or text
text created media or media or created by others
by others are texts created text created are cited with
cited with by others are by others accurate,
accurate, not cited are cited properly
properly with with formatted
formatted accurate, accurate, citations.
citations. properly properly
formatted formatted
citations. citations.
Copyright © Open University Malaysia (OUM)
168 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
Usability The portfolio The portfolio The portfolio The portfolio
and layout is difficult to is often is generally is easy to
read due to difficult to easy to read. read. Fonts
Writing inappropriate read due to Fonts and and type size
convention use of fonts, inappropriate type size vary vary
type size for use of fonts appropriately appropriately
headings, and type size for headings, for headings,
subheadings for headings, subheadings subheadings
and text and subheadings, and text. and text.
font styles text or long
(italic, bold, paragraphs. Use of font
underline). styles is
consistent
Many Some Use of font and improves
formatting formatting styles (italic, readability.
tools are tools are bold,
under or over- under or over- underline) is There are no
utilised and utilised and generally errors in
decrease the decrease the consistent. grammar,
reader readersÊ capitalisation,
accessibility to accessibility to punctuation
the content. the content. and spelling.
There are There are four There are a
more than six or more errors few errors in
errors in in grammar, grammar,
grammar, capitalisation, capitalisation,
capitalisation, punctuation, punctuation
punctuation and spelling and spelling.
and spelling requiring These require
requiring editing and minor editing
major editing revision. and revision.
and revision.
TOTAL
Source: Vandervelde (2018)
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 169
SELF-CHECK 7.6
1. Describe four main steps to develop a portfolio. Which do you
think is the most important step?
2. Examine to what extent portfolio assessments are useful as an
assessment tool.
3. Justify how and when portfolios should be assessed.
ACTIVITY 7.6
Discuss in the myINSPIRE online forum:
(a) To what extent is portfolio assessment used in Malaysian
classrooms?
(b) Do you think portfolio assessment can be used as an assessment
technique in your subject area? Justify your answer.
A project is an activity in which time constraints have been largely removed. It
can be undertaken individually or by a group, and usually involves a
significant element of work done at home or out of school.
A research-based project is more theoretical in nature and may consist of
putting a question, formulating a problem or setting up some hypotheses.
A product-based project would be the production of a concrete object, a
service, a dance performance, a film, an exhibition, play, a computer
programme and so forth.
Project work is a learning experience which enables the development of certain
knowledge, skills and attitudes which prepares students for lifelong learning
and the challenges ahead: knowledge application, collaboration,
communication and independent learning.
Copyright © Open University Malaysia (OUM)
170 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
An effective project should contain the following elements: situation or
problem, project description and purpose, performance specifications, rules,
roles of member and assessment.
The Six AÊs of a project comprises academic rigour, applied learning,
authenticity, active exploration, adult relationships and assessment practices.
Working in groups has become an accepted part of learning as a consequence
of widely recognised benefits of collaborative group work for student learning.
Allocating marks in a project work: include shared group marks, shared-out
marks, individual mark, individual mark (examination) and combination of
group average and individual mark.
A portfolio is a purposeful collection of the works produced by students which
reflects their efforts, progress and achievements in different areas of the
curriculum.
Teachers need to know the benefits and weaknesses of portfolios and use them
to help in studentÊ learning.
A portfolio is a purposeful collection of the work produced by students which
reflects their efforts, progress and achievements in different areas of the
curriculum.
The portfolio provides for continuous and ongoing assessment (i.e. formative
assessment) as well as assessment at the end of a semester or a year (i.e.
summative assessment).
The portfolio assignments can be assessed or graded with a rubric.
As a formative assessment tool, student portfolios can be used by teachers as
informal diagnostic techniques or feedback.
Copyright © Open University Malaysia (OUM)
TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS 171
Artefacts Product-oriented portfolio
Formative assessment Project assessment
Group work Research-based projects
Peer evaluation Rubrics
Portfolio assessment Self-assessment
Portfolios Six AÊs effective projects
Process-oriented portfolio Summative assessment
Product-based project
Bonthron, S., & Gordon, R. (1999). Service learning and assessment: A field
guide for teachers. Evaluation/Reflection. Paper 45. Retrieved from
http://digitalcommons.unomaha.edu/
Bottoms, G., & Webb, L. D. (1998). Connecting the curriculum to „real life.‰
Breaking ranks: Making it happen [Guide ă Non-classroom]. Reston, VA:
National Association of Secondary School Principals.
Bryson, E. (1994). Will a project approach to learning provide children
opportunities to do purposeful reading and writing, as well as provide
opportunities for authentic learning in other curriculum areas? Descriptive
report. ERIC Document No. ED392513.
Chard, S. C. (1992). The project approach: A practical guide for teachers.
Edmonton, Canada: University of Alberta Printing Services.
Edwards, K. M. (2000). EveryoneÊs guide to successful project planning: Tools for
youth. Portland, OR: Northwest Regional Educational Laboratory.
Epstein, A. (2006). Introduction to portfolios. Retrieved from
http://www.teachervision.com
Copyright © Open University Malaysia (OUM)
172 TOPIC 7 PROJECT AND PORTFOLIO ASSESSMENTS
Harwell, S., & Blank, W. (1997). Connecting high school with the real world. ERIC
Document No. ED407586.
Herman, J. L., Aschbacher, P. R., & Winters, L. (1992). A practical guide to
alternative assessment. Alexandria, VA: Association for Supervision and
Curriculum Development.
Katz, L. G., & Chard, S. C. (1989). Engaging the minds of young children: The
project approach. Norwood, NJ: Ablex.
Nitko, A. J. (2001). Educational assessment of students. New Jersey, NJ: Pearson.
Steinberg, A. (1998). Real learning, real work: School-to-work as high school
reform. New York, NY: Routledge.
Sutherland, M. (2003). Peer evaluation checklist for the Biotechnology Academy at
Andrew P. Hill High School. San Jose, CA: East Side Union High School
District.
Paulson F. L., Paulson, P. R., & Meyer, C. (1991). What makes a portfolio a
portfolio? Educational Leadership, 48(1), 60ă63.
Vandervelde, J. (2018). Eportfolio (digital portfolio) rubric. Retrieved from
http://www2.uwstout.edu
Copyright © Open University Malaysia (OUM)
Topic Reliability and
8 Validity of
Assessment
Techniques
LEARNING OUTCOMES
By the end of the topic, you should be able to:
1. Explain the concept of a true score and reliability coefficient;
2. Apply the different methods of estimating the reliability of a test;
3. Compare the different techniques of establishing the validity of a test;
4. Identify the factors affecting reliability and validity; and
5. Discuss the relationship between reliability and validity.
INTRODUCTION
We have discussed the various methods of assessing student performance using
objective tests, essay tests, authentic assessments, project assessments and
portfolio assessment. In this topic, we will address two important issues, namely,
the reliability and validity of these assessment methods. How do we ensure that
the techniques we use for assessing the knowledge, skills and values of students
are reliable and valid? We are making important decisions about the abilities and
capabilities of the future generation and obviously we want to ensure that we are
making the right decisions.
Copyright © Open University Malaysia (OUM)
174 TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES
8.1 WHAT IS RELIABILITY?
What is reliability?
Reliability is the consistency of the measurement.
Let say you gave a Geometry test to a group of Form Five students and one of your
students named Swee Leong obtained a score of 66 per cent in the test. How sure
are you that it is actually the score that Swee Leong should receive? Is that his true
score? When you develop a test and administer it to your students, you are
attempting to measure as far as possible the true score of each student. The true
score is a hypothetical concept with regard to the actual ability, competency and
capacity of an individual. A test attempts to measure the true score of a person.
When measuring human abilities, it is practically impossible to develop an error-
free test and there will be error. However, just because there is an error it does not
mean that the test is not good; what is more important is the size of the error.
Formally, an observed test score, X, is conceived as the sum of a true score, T and
an error term, E. The true score is defined as the average of test scores if a test is
repeatedly administered to a student (and the student can be made to forget the
content of the test in-between repeated administrations). Given that the true score
is defined as the average of the observed scores, so in each administration of a test,
the observed score departs from the true score and the difference is called
measurement error. This statement can be simplified as follows:
Observed score (X) = True score (T) + Error (E)
This departure is not caused by blatant mistakes made by the test writers, but it is
caused by some chance elements in studentsÊ performance on a test. Measurement
error mostly comes from the fact that we have only sampled a small portion of a
studentÊs capabilities. Ambiguous questions and incorrect markings can
contribute to measurement error but it is only a small part of it. Imagine if there
are 10,000 items and a student can obtain 60 per cent if all 10,000 items are
administered (which is not practically feasible). Then, 60 per cent is the true score.
Now, assume that you sample only 40 items to put in a test. The expected score for
the student is 24 items. However, the student may get 20, 26, 30 and so on
depending on which items are in the test. This is the main source of measurement
error. That is, measurement error is due to the sampling of items, rather than
poorly written items.
Copyright © Open University Malaysia (OUM)
TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES 175
Generally, the smaller the error, the greater the likelihood that you are closer to
measuring the true score of a student. If you are confident that your Geometry test
(observed score) has a small error, then you can confidently infer that Swee
LeongÊs score of 66 per cent is close to his true score or his actual ability in solving
geometry problems; i.e. what he actually knows. To reduce the error in a test, you
must ensure that your test is both reliable and valid. The higher the reliability and
validity of your test, the greater the likelihood that you will be measuring the true
score of your students. We will first examine the reliability of a test.
Would your students get the same scores if they took your test on two different
occasions? Would they get approximately the same scores if they took two
different forms of your test? These questions have to do with the consistency of
your classroom tests in measuring studentsÊ abilities, skills and attitudes or values.
The generic name for consistency is reliability. Reliability is an essential
characteristic of a good test because if a test does not measure consistently
(reliably), then you cannot count on the scores resulting from the administration
of the test.
8.2 THE RELIABILITY COEFFICIENT
Reliability is quantified as a reliability coefficient. The symbol used to denote a
reliability coefficient is r with two identical subscripts (for example, rxx). The
reliability coefficient is generally defined as the variance of the true score divided
by the variance of the observed score. The following is the equation.
rxx Variance of the true score 2 true score
Variance of the observed score 2 observed score
Copyright © Open University Malaysia (OUM)
176 TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES
If there is relatively little error, the ratio of the true score variance to the observed
score variance approaches a reliability coefficient of 1.00 which is perfect
reliability. If there is a relatively large amount of error, the ratio of the true score
variance to the observed score variance approaches 0.00, which is total
unreliability (refer to Figure 8.1):
Figure 8.1: Reliability coefficient
High reliability means that the questions of a test tended to „pull together‰.
Students who answered a given question correctly were more likely to answer
other questions correctly as well. If an equivalent or parallel test was developed
by using similar items, the relative scores of students would show little change.
Meanwhile, low reliability indicates that the questions tended to be unrelated to
each other in terms of who answered them correctly. The resulting test scores
reflect that something is wrong with either the items or the testing situation rather
than the studentsÊ knowledge of the subject matter. The following guidelines may
be used to interpret reliability coefficients for classroom tests as shown in
Table 8.1.
Table 8.1: Interpretation of Reliability Coefficients
Reliability Interpretation
0.90 and above Excellent reliability (comparable to the best standardised tests).
0.80ă0.90 Very good for a classroom test.
0.70ă0.80 Good for a classroom test but there are probably a few items which
could be improved.
0.60ă0.70 Somewhat low. There are probably some items which could be
removed or improved.
0.50ă0.60 The test needs to be revised.
0.50 and below Questionable reliability and the test should be replaced or needs major
revision.
Copyright © Open University Malaysia (OUM)
TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES 177
If you know the reliability coefficient of a test, can you estimate the true score of
a student on a test? In testing, we use the standard error of measurement to
estimate the true score.
The standard error of measurement = Standard deviation 1 r
Note: „r‰ is the reliability of the test.
Using the normal curve, you can estimate a studentÊs true score with some degree
of certainty based on the observed score and standard error of measurement.
Example 8.1:
You gave a History test to group of 40 students. Khairul obtained a score of 75 in
the test, which is his observed score. The standard deviation of your test is 2.0.
Earlier, you had established that your History test had a reliability coefficient of
0.7. You are interested in finding out KhairulÊs true score.
The standard error of measurement = Standard deviation 1 r
= 2.0 1 0.7 =2.0 0.55 = 1.1
Therefore, based on the normal distribution curve (refer to Figure 8.2), KhairulÊs
true score should be:
(a) Between 75 ă 1.1 and 75 + 1.1 or between 73.9 and 76.1 for 68 per cent of the
time.
(b) Between 75 ă 2.2 and 75 + 2.2 or between 72.8 and 77.1 for 95 per cent of the
time.
(c) Between 75 ă 3.3 and 75 + 3.3 or between 71.7 and 78.3 for 99 per cent of the
time.
Copyright © Open University Malaysia (OUM)
178 TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES
Figure 8.2: Determining KhairulÊs true score based on a normal distribution
SELF-CHECK 8.1
1. Define the reliability of a test.
2. What does the reliability coefficient indicate?
3. Explain the concept of a true score.
ACTIVITY 8.1
Shalin obtains a score of 70 in a Biology test. Given the reliability of the
test is 0.65 and the standard deviation of the test is 1.5. The teacher was
planning to select students who had scored 70 and above to take part in
a Biology competition. The teacher was not sure whether he should select
Shalin since there could be an error in her score. Should he select Shalin?
Why?
Post your answer on the myINSPIRE online forum.
(Use the standard error of measurement.)
Copyright © Open University Malaysia (OUM)
TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES 179
8.3 METHODS TO ESTIMATE THE RELIABILITY
OF A TEST
Let us now discuss how we estimate the reliability of a test. Figure 8.3 lists three
common methods of estimating the reliability of a test. It is not possible to calculate
reliability exactly and so we have to estimate reliability.
Figure 8.3: Methods for estimating reliability
These three methods are further explained as follows:
(a) Test-retest
Using the test-retest technique, the same test is administered again to the
same group of students. The scores obtained in the first administration of the
test are correlated to the scores obtained on the second administration of the
test. If the correlation between the two scores is high, then the test can be
considered to have high reliability. However, a test-retest situation is
somewhat difficult to conduct as it is unlikely that students will be prepared
to take the same test twice.
There is also the effect of practice and memory that may influence the
correlation. The shorter the time gap, the higher the correlation; the longer
the time gap, the lower the correlation. This is because the two observations
are related over time. Since this correlation is the test-retest estimate of
reliability, you can obtain considerably different estimates depending on the
interval.
(b) Parallel or Equivalent Forms
For this technique, two equivalent tests (or forms) are administered to the
same group of students. The two tests are not similar but are equivalent. In
other words, they may have different questions but they are measuring the
same knowledge, skills or attitudes. Therefore, you have two sets of scores
which are correlated and reliability can be established. Unlike the test-retest
technique, the parallel or equivalent forms reliability measure is not affected
Copyright © Open University Malaysia (OUM)
180 TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES
by the influence of memory. One major problem with this approach is that
you have to be able to generate a lot of items that reflect the same construct.
This is often not an easy feat.
(c) Internal Consistency
Internal consistency is determined using only one test administered once to
the students. Internal consistency refers to how the individual items or
questions behave in relation to each other and the overall test. In effect, we
judge the reliability of the instrument by estimating how well the items that
reflect the same construct yield similar results. We are looking at how
consistent the results are for different items for the same construct within the
measure. The following are two common internal consistency measures that
can be used.
(i) Split-half
To solve the problem of having to administer the same test twice, the
split-half technique is used. In this technique, a test is administered
once to a group of students. The test is divided into two equal halves
after the students have completed the test. This technique is most
appropriate for tests which include multiple-choice items, true-false
items and perhaps short-answer essays. The items are selected based
on odd-even method whereby one half of the test consists of odd
numbered items, while the other half consists of even numbered items.
Then, the scores obtained for the two halves are correlated to determine
the reliability of the whole test using the Spearman-Brown correlation
coefficient.
2rxy
1 rxy
rsb
In this formula, rsb is the split-half reliability coefficient and rxy
represents the correlation between the two halves. Say for example, you
have established that the correlation coefficient between the two halves
is 0.65. What is the reliability of the whole test?
2rxy 20.65 1.3 0.79
1 rxy
rsb 1 0.65 1.65
Copyright © Open University Malaysia (OUM)
TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES 181
(ii) CronbachÊs Alpha
CronbachÊs coefficient alpha can be used for both binary-type
(1 = correct, 0 = incorrect or 1 = true and 0 = false) and scale items
(1 = strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree).
Reliability is estimated by computing the correlation between the
individual questions and the extent to which individual questions
correlate with the total test. This is meant by internal consistency. The
key is „internal‰, unlike test-retest and parallel or equivalent form that
require another test as an external reference. The stronger the items are
inter-related, the more likely the test is consistent. The higher the alpha,
the more reliable is the test. There is no generally agreed cut-off point.
Usually, 0.7 and above is acceptable (Nunnally, 1978). The formula for
CronbachÊs alpha is as follows:
k pi 1
CronbachÊs i 1 pi
k
alpha () K 1 1 2
x
where,
k is the number of items in the test;
pi refers to item difficulty which is the proportion of students who
answered the item i correctly; and
2x is the sample variance for the total score.
Example 8.2:
Suppose that in a multiple-choice test consisting of five items or
questions, the following difficulty index for each item was observed:
p1 = 0.4, p2= 0.5, p3 = 0.6, p4 = 0.75 and p5 = 0.85. Sample variance
(2x) = 1.84. CronbachÊs alpha would be calculated as follows:
5 5 1 1.045 0.54
1 1.840
Copyright © Open University Malaysia (OUM)
182 TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES
Professionally developed standardised tests should have an internal
consistency coefficient of at least 0.85. High reliability coefficients are
required for standardised tests because they are administered only
once and the score on that one test is used to draw conclusions about
each studentÊs ability level on the construct measured. Perhaps, the
closest to a standardised test in the Malaysian context would be the
tests for different subjects conducted at the national level in the PMR
and SPM.
According to Wells and Wollack (2003), it is acceptable for classroom
tests to have reliability coefficients of 0.70 and higher because a
studentÊs score on any one test does not determine the studentÊs entire
grade in the subject or course. Usually, grades are based on several
other measures such as project work, oral presentations, practical tests,
class participation and so forth. To what extent is this true in the context
of the Malaysian classroom?
A Word of Caution!
When you get a low alpha, you should be careful not to immediately conclude that
the test is a bad test. Instead, you should check to determine if the test measures
several attributes or dimensions rather than one attribute or dimension. If it does,
there is the likelihood for the CronbachÊs alpha to be deflated.
For example, an aptitude test may measure three attributes or dimensions such as
quantitative ability, language ability and analytical ability. Hence, it is not
surprising that the CronbachÊs alpha for the whole test may be low as the questions
may not correlate with each other. Why? This is because the items are measuring
three different types of human abilities. The solution is to compute three different
CronbachÊs alphas ă one for quantitative ability, one for language ability and one
for analytical ability.
SELF-CHECK 8.2
1. What is the main advantage of the split-half technique over the test-
retest technique in determining the reliability of a test?
2. Explain the parallel or equivalent form technique in determining
the reliability of a test.
3. Explain the concept of internal consistency reliability.
Copyright © Open University Malaysia (OUM)
TOPIC 8 RELIABILITY AND VALIDITY OF ASSESSMENT TECHNIQUES 183
8.4 INTER-RATER AND INTRA-RATER
RELIABILITY
Whenever you use humans as a part of your measurement procedure, you have to
be concerned whether the results you get are reliable or consistent. People are
notorious for their inconsistency. We are easily distracted. We get tired of doing
repetitive tasks. We daydream. We misinterpret. Therefore, how do we determine
whether:
(a) Two observers are being consistent in their observations?
(b) Two examiners are being consistent in their marking of an essay?
(c) Two examiners are being consistent in their marking of a project?
Let us analyse these problems from the perspectives of:
(a) Inter-rater Reliability
When two or more people mark essay questions, the extent to which there is
agreement in the marks allotted is called inter-rater reliability (refer to
Figure 8.4). The greater the agreement, the higher is the inter-rater reliability.
Figure 8.4: Examiner A versus Examiner B
Copyright © Open University Malaysia (OUM)
The words you are searching are inside this book. To get more targeted content, please make full-text search by clicking here.
HMEF5053 Measurement and Evaluation in Education_vDec19
Discover the best professional documents and content resources in AnyFlip Document Base.
Search