E-proceeding PEERS'22

feedback, flexibility, learner activity, the value of knowledge, goal
orientation, experiential value, knowledge organisation, and metacognition.
The evaluation is based on the Likert scale ranging from 1 to 4, in which
each scale has a different description.

3.7 Learnability

Learnability is a quality of goods and interfaces which allows users
to rapidly become acquainted with them and fully utilise all their features
and capabilities. According to Spierling (2018), learnability is the
understanding allowing users to evaluate everything else as their learning
curve. The parameters used in evaluating learnability are based on the seven
(7) elements of digital storytelling (Yuksel-Arslan et al., 2016). The
parameters are the point of view, dramatic question, emotional content,
economic resources, the use of voice, soundtrack, and pacing. Furthermore,
using a Likert scale ranging from 1 to 4, each value has been explained
following the specifications.

3.8 Reliability

Validity and reliability are two critical aspects to consider when
evaluating a measurement system. The consistency of measurement
outcomes is what reliability is all about (Shiratuddin & Kuan, 2014).
According to Yamac & Ulusoy (2016), the “Digital Story Assessment
Rubric” was used to evaluate the quality of the digital stories created by the
students. The parameters used in evaluating the reliability of the digital
storytelling applications are drama question, emotion, pictures and videos,
audio and music, voice, novelty and creativity, economy, pacing, point of
view, setting, characters, plot, and aim of the story. The evaluation is based
on a Likert scale of 1 to 4, with various descriptions for each scale.

4. Discussion

Many features of the classroom environment and activities allowing
digital storytelling integration and enhance engaged learning have been
discovered through observations and interviews (Sadik, 2008). According
to the findings on the use of digital narrative creation as a teaching tool, it is
an excellent tool, strategy, or approach for the learning and teaching process

of children growing up with technology. However, the users will be curious
to use the digital storytelling application to determine whether the
application is good for learning tools.

When a digital story is used as a teaching tool, it will first capture
the students’ interest. Thus, utilising technological resources will make them
eager to learn, transforming them into active participants in their internalised
projects (Seker, 2016). The factors involved in evaluating the digital
storytelling will be used to decide whether the application satisfies the users’
needs.

Thus, to evaluate digital storytelling, an evaluation model is needed.
However, some is lacking in implementing the Kirkpatrick Model and CIPP
Model. As stated above, both evaluation models involved four (4) steps. The
Kirkpatrick Model involves reaction, learning, behaviour, and results.
Meanwhile, the CIPP Model involves context, input, process, and product.
Supposedly, additional parameters are needed to be evaluated if digital
storytelling is a part of learning tools. The parameters that need to be taken
into consideration are usability, effectiveness, learnability, and reliability.
This is because digital storytelling involves technology not only in content
but also in the whole application. Therefore, these additional parameters are
needed in the digital storytelling evaluation model to evaluate digital
storytelling as a learning tool.

4.1 The Comparison of Evaluation Models

Table 1 shows the comparison between the Kirkpatrick Model,
CIPP Model and DSEM. The comparison includes several properties such
as purpose, interaction of the user, involvement of user in evaluation,
include formative and summative and description for each model. Each
model has its own characteristics and specialities.

Generally, the digital storytelling involves user experience such as
commitment, involvement, emotion, and user motivation where it requires
user to interact directly with the application. DSEM is a proposed model
adapted from Kirkpatrick Model and CIPP Model. It inherits some processes
that can be used in conducting the evaluation on the digital storytelling as a
learning tool. Since the Kirkpatrick Model involves interaction and
experience in the training and development programme, some of the
evaluation needed from the formative and summative aspects is used to

evaluate the effectiveness of the deliverable content. Formative and
summative factors are very important in the CIPP Model to assess the
programme content or the curriculum syllabus.

5. Conclusion

This study aims to show how to evaluate a digital storytelling
application. Several evaluation attributes have been discovered from the
study, and each has different parameters, which will be used in evaluation
processes.

Digital storytelling can be applied in entertainment and learning.
Concerning learning, we need to know how beneficial digital storytelling
can be. We need to know how it can improve the learning style of the
teachers and the students. Digital storytelling can also be used in learning
curriculum in which it helps teachers plan the courseware and attracts
students in their studies. Thus, an appropriate evaluation model is needed to
evaluate the digital storytelling application as a learning tool. Before this,
teachers usually create their own stories in which they need to plot a story
to attract the students, but now, with digital storytelling, the teachers can use
it for learning purposes. Furthermore, multimedia, such as video, image, and
audio, can attract students to more focus in the class.

Hence, there is a need to encourage teachers to provide students
with additional long-term and problem-solving chances to spend enough
time working and thinking together and creating and presenting their digital
stories. Digital storytelling in learning can be innovated if there is a good
evaluation model. In the future, a DSEM will be developed to evaluate the
credibility of the digital storytelling application as a learning tool.

Table 1. The Comparison between the Kirkpatrick Model, CIPP Model

and DSEM

Properties Kirkpatrick Model CIPP Model DSEM

Purpose To evaluate training To evaluate school To evaluate
and development curriculums and digital storytelling
Involve the programme. courseware. that will be used
interaction of the as a teaching tool.
user Yes No
Yes

Involvement of Yes No Yes

user in the
evaluation

Formative / No Yes Yes
summative
Considerably aided in In the spirit of The model is used
Description the advancement of improving training to evaluate the
training assessment assessments, it is digital storytelling
thoughts and practise. not only focused on as a teaching tool.
programme
This model includes a outcomes, but also It is the adaptation
method for displaying on continuous from Kirkpatrick
training outcomes as improvement and Evaluation Model
well as various data accountability, cost, and CIPP
that may be utilised and programme Evaluation Model.
for evaluation. needs.
To identify
It facilitates the The concept is built whether the
difficult task of on "learning by digital storytelling
evaluating training doing," which is a is suitable to be
programmes. constant attempt to used as a learning
detect and fix errors tool or not.
produced in
evaluation practise,
as well as a
technique to create
and test new
processes for more
successful practises.

7. Acknowledgements

The authors would like to express appreciation for the support of
the research grant 600-RMC/GPK 5/3 (046/2020) from Universiti
Teknologi MARA (UiTM), Malaysia.

8. References

Akbari, M., Dorri, S., & Sedeh, M. (2016). Kirkpatrick evaluation model for

in-service training on cardiopulmonary resuscitation. Iranian

Journal of Nursing and Midwifery Research, 21(5), 493.

https://doi.org/10.4103/1735-9066.193396

Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick’s four-

level model of training criteria to evaluate training programmes for

head teachers. Education Sciences, 11(3), 116.

https://doi.org/10.3390/educsci11030116

Anh, V. T. (2018). Evaluation models in educational program: Strengths and

weaknesses. VNU Journal of Foreign Studies, 34(2).

https://doi.org/10.25073/2525-2445/vnufs.4252

Aziz, S., Mahmood, M., & Rehman, Z. (2018). Implementation of CIPP

model for quality evaluation at school level: A case study. Journal

of Education and Educational Development, 5(1), 189.

https://doi.org/10.22555/joeed.v5i1.1553

Bashri, A., Prastiwi, M. S., & Puspitawati, R. P. (2020). CIPP model for

curriculum evaluation of biology education. Proceedings of the

International Joint Conference on Arts and Humanities (IJCAH

2020). https://doi.org/10.2991/assehr.k.201201.209

Eck, J. (2006). An Analysis of the Effectiveness of Storytelling with Adult

Learners in Supervisory Management [Master’s thesis].

Gimeno-Sanz, A. (2015). Digital storytelling as an innovative element in

English for specific purposes. Procedia - Social and Behavioral

Sciences, 178, 110-116.

https://doi.org/10.1016/j.sbspro.2015.03.163

Houston, D., & Thompson, J. N., (2017). Blending formative and

summative assessment in a capstone subject: ‘It’s not your tools, it’s

how you use them’, Journal of University Teaching & Learning

Practice, 14(3), 2017. Available

at:http://ro.uow.edu.au/jutlp/vol14/iss3/2.

Jankins, M., & Lonsdale, J. (2007). Evaluating the effectiveness of digital

storytelling for student reflection. Proceedings ascilite Singapore

2007.

Kirkpatrick model: Four levels of learning evaluation. (2018, September 6).

Educational Technology.

https://educationaltechnology.net/kirkpatrick-model-four-levels-

learning-evaluation/

Ohler, J. B. (2013). Digital storytelling in the classroom: New media

pathways to literacy, learning and creativity. Corwin Press

Paull, M., Whitsed, C., & Girardi, A. (2016). Applying the Kirkpatrick

model: Evaluating an Interaction for Learning Framework

curriculum intervention. Issues in Educational Research, 26(3).

Poblete, M. O. (2014). Using the CIPP Model to Assess the University’s

Health Programs for Faculty and Staff. CNU Journal of Higher

Education, 8, 77-89.

Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A critique of

Kirkpatrick’s evaluation model. New Horizons in Adult Education

and Human Resource Development, 29(2), 35-53.

https://doi.org/10.1002/nha3.20178

Sadik, A. (2008). Digital Storytelling: A meaningful Technology-Integrated

Approach for Engaged Student Learning. Education tech Research

Dev. https://doi.org/10.1007/s11423-008-9091-8

Seker, B. S. (2016). An Evaluation of Digital Stories Created for Social

Studies Teaching. Journal of Education and Practice, 7(29), 2222-

1735.

Shiratuddin, N., & Kuan, T. H. (2014). Quality evaluation of a digital

storytelling (DST) conceptual model. 2014 International

Conference on Multimedia Computing and Systems (ICMCS).

https://doi.org/10.1109/icmcs.2014.6911376

Spierling, U. (2018). Tools and Principles for Creation in Interactive

Storytelling: The Issue of Evaluation.

Wagner, S., & Deissenboeck, F. (2019). Defining productivity in software

engineering. Rethinking Productivity in Software Engineering, 29-

38. https://doi.org/10.1007/978-1-4842-4221-6_4

Wu, P., Hu, H., Wu, F., & Fan, K. (2017). The evaluation on the usability

of digital storytelling

teaching system in teaching. Learning and Collaboration Technologies.

Technology in Education, 473-487. https://doi.org/10.1007/978-3-

319-58515-4_36

Yamac, A., & Ulusoy, M. (2016). The Effect of Digital Storytelling in
Improving the Third Graders’ Writing Skills*. International
Electronic Journal of Elementary Education, 9(1), 59-86

Yuksel-Arslan, P., Yildirim, S., & Robin, B. R. (2016). A phenomenological
study: Teachers’ experiences of using digital storytelling in early
childhood education. Educational Studies, 42(5), 427-445.
https://doi.org/10.1080/03055698.2016.1195717