Research Insights in Teaching and Learning

In addition, the students consistently demonstrated features of CEFR C1 abilities by showing excellent grammatical control and accuracy throughout their presentations. Errors were rare and when they made errors, they were able to self-correct themselves immediately to ensure that their points were delivered clearly. In other words, the students were consistently maintaining a high degree of grammatical accuracy regardless of the types of sentences they were producing, and they too displayed accuracy in the application of other grammatical rules such as tenses, pronouns, referents, plural forms and word forms. Clearly, these abilities and performance are in line with the descriptors for CEFR C1 level. Finally, all the students could deliver their presentations smoothly, fluently and effortlessly. It is interesting to note that the use of fillers was rare and if there were any, they were often insignificant and unnoticeable. The quality of their presentations was enhanced by consistently clear and intelligible pronunciation with appropriate word and sentence stress which reflects that the students’ spoken proficiencies were at CEFR C1 level. Features of CEFR C1 proficiency level were also apparent with regard to coherence, whereby all of the students managed to maintain a smooth and coherent flow from the beginning to the end of their presentations. Moreover, the progression from the introduction and thesis statement to the elaborations of main ideas in the body and finally to the concluding section was effectively enhanced by the appropriate and effective use of various transitional signals. These, without a doubt illustrates that the students are at CEFR C1 level in terms of giving oral presentations. The findings of this study confirms that the course learning outcome with regard to oral presentation has been attained and that the students showed the performance and abilities that are aligned with the CEFR C1 descriptors. The findings are however different from the findings of a study conducted by Razali and Abd Latif (2019). Even though both studies were carried out among students in public universities in Malaysia, we observe that unlike in the previous study where the students considered themselves to be either CEFR B1 or B2, the present study suggests that the group of students in the EAW course at the IIUM are at CEFR C1. The difference could be because English is the medium of instruction and communication at the IIUM, as such, it is expected that the students fare better in their spoken abilities. In the same vein, compared to the study conducted by Waluyo (2019) among first year students in Thailand, there seems to be a big gap between the oral proficiency level of the Thai students (majority CEFR A1 and A2) and the students in the present study (CEFR C1). Such stark contrast could be due to the exposure to English received by the students in Thailand and in Malaysia. Students in Malaysia have a strong foundation in the English language because English being a second language is widely spoken in Malaysia. In Thailand, on the other hand, the use of English is not as extensive because it is considered as a foreign language. Hence, that explains why students in Malaysia generally have a good mastery of spoken English. 42

Conclusion Based on the findings and discussion, it can be concluded that the three students selected for this study showed the abilities to give a clear, well-structured presentation of a complex subject and had successfully provided elaborations using reasons, supporting details and examples. It is important to note that these are the abilities that are included in the course’s learning outcome for speaking skill which has also been mapped to the Can-do statement for sustained monologue: addressing audience or giving presentations at CEFR C1 level as shown in Table 1. Thus, this demonstrates that the learning outcome for the course’s speaking skill has indeed been attained. In addition, the students also exhibited the abilities of CEFR C1 level in terms of range, accuracy, fluency, coherence and phonology as shown in Table 2. Choosing a learning outcome for addressing audience (giving a presentation) that is pitched at CEFR C1 level is important for undergraduates so that they can perform well academically. This is because studies have shown that students with high spoken proficiency perform better in spoken tasks and assessments as well as have higher academic than students with low proficiency (Aina et.al., 2013; Liu and Jia, 2017). Nevertheless, since this case study only focused on three students who were randomly selected, this may not represent the attainment of the learning outcome among all students who registered for the course. Therefore, it is recommended that an investigation be carried out involving a bigger number of students so that we can gain more accurate confirmation. There might be students who have not attained the learning outcome; hence, to bring students up to the desired proficiency level, instructors can conduct exciting meaningful activities for the students to boost their confidence and independence. Students need positive surroundings to enhance their speaking abilities. It is also crucial that the course instructors create activities that enable students to hone the abilities stated in the learning outcome for the learning outcome to be attained at the end of the semester. 43

References Aina, J.K, Ogundele A. G. & Olanipekun, S. S. (2013). Students’ Proficiency in English Language Relationship with Academic Performance in Science and Technical Education. American Journal of Educational Research, 2013, 1 (9), 355-358. Council of Europe, (2020). Common European Framework of Reference for Languages: Learning, Teaching Assessment. Companion Volume. English Language Education Reform in Malaysia: The Roadmap 2015-2025. (2015). Ministry of Education Malaysia. ISBN: 978-967-5387-45-6. https://anyflip.com/detl/zspi/basic Foley, J. A. (2019). Issues on assessment using CEFR in the region. LEARN Journal: Language Education and Acquisition Research Network Journal, 12(2), 28-48. https://files.eric.ed.gov/fulltext/EJ1225665.pdf Liu, L. & Jia, G. (2017). Looking beyond scores: Validating a CEFR-based university speaking assessment in mainland China. Language Testing in Asia, 7(2). DOI 10.1186/s40468-107-0034-3. https://languagetestingasia.springeropen.com/articles/10.1186/s40468-017-0034-3 Mohd Don, Z. & Abdullah, M. H. (2019, May 22). The reform of English language education in Malaysia. Free Malaysia Today. https://www.moe.gov.my/menumedia/media-cetak/keratan-akhbar/the-reform-ofenglish-language-education-in-malaysia-free-malaysia-today-22-mei-2019 Razali, N. H. & Abdul Latif, L. (2019). CEFR-based English speaking skill self-assessments by Malaysian graduating non-native English speaking students. Malaysian International Journal of Research in Teacher Education, 2, 82-93. ISSN: 2289-8808, eISSN: 7210-7132. https://mijoriteitepc.wixsite.com/index Waluyo, B. (2019). Thai first year university students’ English Proficiency on CEFR Levels: A case Study of Walailak University, Thailand. The New English Teacher. Institute for Language Education Assumption University, 13, 51-71. ISSN: 1905-7725. https://www.researchgate.net/publication/335079291_Thai_FirstYear_University_Students'_English_Proficiency_on_CEFR_Levels Wanna, W. Tilahun, A. & Pawlos, A. (2016). The English language proficiency of first year students in Dilla University. The Ethiopian Journal of Education, XXXVI (1), 111-147. http://ejol.aau.edu.et/index.php/EJE/article/view/88 44

The impact of CEFR implementation on language education in Malaysia Kulwant Kaur a/p Kartar Singh English Language Division, Centre for Languages and Pre-University Academic Development (CELPAD), International Islamic University Malaysia, Gombak (E-mail: [email protected]) Abstract The Common European Framework of Reference for Languages (CEFR) has been widely used as a global standard for language teaching and learning in Malaysia since its establishment. The CEFR framework strives to lay the foundation for language proficiency, encompassing language curriculum, reference materials and assessments. Despite gaining popularity in modern language education, its comprehensive impact on language classroom pedagogy at the university level remains largely unexplored. Thus, this research seeks to investigate the implementation of CEFR in universities and its effects on language education. Six university lecturers from two different universities participated in this study. Interview sessions were carried out to find out what their perceptions on the implementation of CEFR in their classrooms are and to identify the challenges they faced in incorporating CEFR framework when teaching the English language. The analysis showed that even though there were challenges such as effects on instructors’ autonomy and insufficient local CEFR-aligned textbooks in the journey to implement CEFR, positive effects were identified where comprehending the framework as well as students’ learning processes were concerned. The participants admitted to be familiar with the six levels of CEFR global scale due to comprehensive training and exposure to the framework. The participants also displayed a strong dedication to acquiring new knowledge and showcased their ability to quickly grasp concepts. They asserted a significant familiarity with the CEFR global scale. Keywords: CEFR, language curriculum, challenges, teaching, learning Introduction The Common European Framework of Reference for Languages, known as CEFR, was established in 2001 with the aim of creating global standards for foreign language education. It caters to the need of language learners, academics, and professionals involved in language assessment and teaching and learning. CEFR provides comprehensive guidelines on the language proficiency that learners should achieve for effective communication in a language. It also serves as a widely recognized framework for the mutual recognition of language qualifications, aiding learners, teachers, course designers, examining bodies, and educational administrators in coordinating their efforts. 46 1 1

The framework comprises six levels of descriptors, categorizing language learners into three main groups: Proficient users (levels C1 & C2), Independent users (levels B1 & B2) and Basic users (levels A1 & A2). The “can do” statements for listening, writing, reading and speaking skills offer detailed descriptions of learners’ abilities at each level. The implementation of CEFR in Malaysia commenced with the establishment of the English Language Standards and Quality Council (ELSQC) in 2013. The council collaborated with the English Language Teaching Center (ELTC) to enhance the English language proficiency of Malaysian students, with a focus on aligning the education system with the CEFR framework. A roadmap for systematic reforms in English language education was developed to elevate the standard of education to international levels, encompassing preschool through tertiary education. It is important to note that the adoption of CEFR in Malaysia is in its early stages, marking the beginning of the second wave, which is still regarded as a novel development. It is challenging to determine the success or failure of CEFR integration in the education system at this stage, as it is still too early to assess its impact in a short period. However, previous studies have highlighted anticipated and unforeseen problems that may arise during the implementation of CEFR in Malaysia. Hence, conducting a local study focusing on CEFR is crucial to address the awareness and familiarity concerns as it is important to find out how English language instructors comprehend and grasp the challenges tied to incorporating a CEFR-aligned curriculum into their teaching and learning activities. This paper aims to answer the following research questions: 1.What are university lecturers’ perceptions in adopting the CEFR-aligned curriculum? 2.What consequences arise from the implementation of the CEFR at the university level? Literature Review The sudden inclusion of CEFR in the English language curriculum has led some teachers to become aware and prepared to implement the revised curriculum. However, certain teachers expressed their concerns about challenges related to integrating CEFRaligned lessons and assessments into their current teaching, learning, and assessment 47

methods. It is natural for teachers to react this way when faced with policy changes, as such changes often bring about confusion and resistance. Some teachers find it difficult to embrace these changes. As a result, it is advisable for teachers to stay informed and proficient in order to effectively embrace and adjust to changes in teaching and learning approaches over time (Jerald & Shah, 2018). Interpreting a curriculum is an intricate process that goes beyond simply reading the document and putting it into action. This process requires teachers to understand it deeply, and factors like their beliefs, thoughts, perceptions, and other variables can influence how they execute the curriculum and its impact on stakeholders. Teachers’ perceptions pertaining to CEFR comprehension is mandatory as they are the ones who are going to integrate CEFR into their teaching and learning processes. Perception of teachers can be regarded as the comprehension, perspective, and behavior exhibited in response to the integration of a curriculum aligned with CEFR standards (Mohamad Khair & Mohd Shah, 2021). Their perceptions stem from their practical encounters, expertise, and educational background, all of which mirror their preparedness and willingness to implement the new CEFR-aligned curriculum in accordance with its aims and anticipated outcomes. Significant discoveries regarding teachers' apprehension and unease concerning the adoption of CEFR in Malaysia underscore the challenges faced by English educators in the country. These challenges stem from inadequate CEFR training, ineffective instructional sessions, and a lack of both preparedness and acceptance (Mohamad Uri, 2021). Regrettably, the available CEFR-related training sessions and workshops remain insufficient and deficient. The issue of attending a solitary CEFR training session is already a concern, with the only workshop attended being one aimed at acquainting teachers with the framework (Mohamad Uri & Abdul Aziz, 2018). Evidently, these teachers necessitate more comprehensive training, especially pertaining to the integration of the framework into their teaching methodologies. A single workshop with the sole purpose of introducing and acquainting teachers with the surface-level concepts of CEFR is evidently inadequate. Several studies have concentrated on the implementation of CEFR and its effects on English language education in Malaysia. One such study, conducted by Mohamad Uri and Abdul Aziz (2018), underscored the challenges encountered during CEFR implementation in 48

Malaysia. This study aimed to identify key issues faced by Malaysian Ministry of Education (MOE) in executing CEFR. Notable challenges were highlighted including a significant number of English teachers who lacked proficiency, English being the weakest subject in national assessments for students, limited political support, insufficient teacher training, and a dearth of research on CEFR implementation within the Malaysian context. Mohamad Uri and Abdul Aziz (2018) subsequently conducted another study, focusing on the awareness and challenges faced by English secondary school teachers and Ministry of Education officials in implementing CEFR. The study employed questionnaires and interviews, revealing that most teachers possessed limited knowledge, minimal exposure, and low awareness of CEFR, yet they remained optimistic about its importance in enhancing English proficiency. Meanwhile, officials expressed confidence that with adequate time and training, adoption would yield positive results despite obstacles such as teacher resistance, inadequate training and negative perceptions. Significantly, teachers exhibited high levels of concern and anxiety regarding CEFR implementation due to uncertainty about their roles and insufficient information and preparation regarding the reform (Lo, 2018). Specifically examining the concerns of Form 1 and Form 2 English teachers regarding CEFR adoption, Lo’s (2018) study revealed heightened apprehension about various aspects, including awareness, personal impact, management, and consequences of implementation. In the tertiary context, Darmie et al. (2017) examined teachers' perspectives on students' performance in English Language Proficiency courses using CEFR descriptors. Findings indicated diverse viewpoints among teachers regarding students' performance in English language proficiency courses at a public university in Malaysia. Noteworthy studies on assessment include those by Sidhu et al. (2018) and Sabbir (2019). Overall, these studies reported generally positive opinions on CEFRaligned assessment among teachers. Nevertheless, challenges emerged, such as limited understanding of the methodology, insufficient knowledge of the CEFR-aligned ESL curriculum, resource shortages, time constraints, classroom size, heavy workloads, and inadequate training in effectively implementing the CEFR-aligned curriculum and assessment (Mohd Khair & Mohd Shah, 2021). A specific study focused on English language teachers’ perceptions of CEFR-aligned curriculum implementation in Malaysian primary schools delved into teachers’ perspectives, 49

identified challenges, and proposed viable solutions to CEFR implementation. Findings emphasized that many teachers had limited knowledge and exposure to CEFR, yet remained hopeful and confident in the importance of CEFR for improving English proficiency. Additionally, challenges such as inadequate training, non-local textbooks, insufficient ICT support and resources were noted. Another local study conducted by Jerald and Shah (2018) sought insights from English teachers regarding the impact of the CEFR-aligned Curriculum on ESL instruction. Findings indicated overwhelmingly positive feedback from teachers, which could serve as a valuable basis for relevant authorities and stakeholders to assess the validity and reliability of the CEFR-aligned curriculum as a robust platform. Methods Research Design and Methods This study employed a qualitative research design, aiming to delve deeper into the perspectives of participants regarding CEFR implementation in English language classrooms Malaysia. Specifically, a semi-structured interview format was chosen to gain comprehensive insights from teachers about their experiences with implementing the CEFR in their classrooms. Conducting interviews proved highly beneficial in comprehending participants' perceptions and attitudes toward this phenomenon. In the context of this research, the semistructured interview played a crucial role in uncovering the participants' viewpoints on the implementation of CEFR in their educational settings. Interview questions that aligned with the research goals were constructed. An interview protocol consisting of questions including inquiries about participants’ demographic background, viewpoints about the integration of CEFR framework with the English language curriculum and challenges encountered were administered to the participants. The interview sessions were held face to face and interview session for each participant was held for approximately 2 hours to gain detailed feedback regarding their perceptions and the impact of CEFR on the teaching and learning of the English language. Participants This study engaged 6 university lecturers from 2 universities in Selangor Darul Ehsan. Lecturers teaching the English language were chosen through criterion sampling due to their direct involvement with and impact on CEFR. The participants chosen were experienced lecturers who had taught for fifteen to thirty years. These lecturers play a pivotal role in introducing, implementing and integrating CEFR into their teaching methods. Additionally, they serve as gatekeepers, influencing the success or failure of implemented policies. Therefore, their perspectives on CEFR’s implementation in Malaysia are invaluable, aiding policymakers in evaluating policy outcomes. 50

Findings/Results The detailed outcomes of applying CEFR to language education in Malaysia were meticulously documented through semi-structured interviews. These sessions gathered data, which was then categorized into two parts aligning with the research questions. Consequently, the discussion of the results corresponds with the outlined research inquiries. What are university lecturers’ perceptions in adopting the CEFR-aligned curriculum? The feedback garnered from participants concerning the adoption of CEFR reflects a positive outlook, as evident from the interview sessions. The participants widely endorse the incorporation of CEFR into the syllabus and assessments, citing a global perspective. They agreed that the integration of CEFR aligns with the concept of globalization. CEFR integration enhances students’ readiness for their future as the inclusion of CEFR contributes to the preparation of university students for the challenges of post-university life. It is perceived as instrumental in elevating English proficiency among Malaysians and enhancing competitiveness globally. This aligns with the overarching goal of enhancing students’ readiness for the global workforce. CEFR was also believed to boost English proficiency levels. One participant said “We don’t want to be left behind as every other country is integrating CEFR in their curriculum”. CEFR is already being extensively utilized by neighboring countries such as Indonesia, the Philippines, and Vietnam. Hence we have to align Malaysia’s approach with the prevailing global practices to be on par with them. 51

It was also addressed by the participants that CEFR will be effective for the persistently low English proficiency levels among Malaysian students. The CEFR framework that is standardized could elevate the recognition and global acceptance of Malaysian education system standards. A clear and universally understood benchmark for Malaysian students’ language proficiency will hence be comparable with other countries. The apparent shift from teacher-centric, knowledge-focused classrooms to learnercentered, communication-oriented instruction is the main benefit of employing "can do" phrases in pedagogy. Although the change cannot be entirely credited to the CEFR, the participants are confident that language educators and learners will benefit from this transformation if the techniques and concepts indicated by the CEFR are diligently adopted. CEFR also has an impact on self-regulated learning. Nowadays, a lot of teachers understand that for learning to have an impact, students need to be accountable for their own education. One of the participants was quoted saying, “students are responsible for their own learning and the can do statements play a significant role in this". These “can do" statements, which express goal-setting and assessment, have been widely accepted in settings ranging from individual classes to entire institutions. Thus, the continuous transition to learner-centered instruction will continue in spite of the fact that the CEFR might not be widely familiar or explicitly acknowledged among a considerable cohort of educators in Malaysia. What consequences arise from the implementation of the CEFR at the university level? The participants believed that a potential obstacle hindering the smooth execution of CEFR implementation in Malaysia could be attributed to the lecturers themselves. This sentiment likely stems from their limited preparation to fully embrace the novel framework. Their constrained comprehension and limited exposure to CEFR, coupled with the rapid implementation process, could contribute to their reluctance to accept and seamlessly incorporate the framework. This may be attributed to the fact that these educators were not adequately equipped to handle the new approach. As one of the participant was quoted saying, “Educators were not adequately quipped to handle the new approach”. Another participant claimed that “CEFR is very student-centered and it has changed classroom practices”. 52

She also added that, “Teachers could be a hindrance to CEFR implementation in Malaysia”. It is noteworthy to point out that while a significant portion (2 participants) perceived CEFR as a potential challenge, a substantial portion (4 participants) held a contrasting viewpoint as pointed out by one participant, “I think we are fully aware of the importance of CEFR in education. The university has stepped up its efforts to fully prepare teachers for the new approach and I think it has done an excellent job”. The participants concurred that the limited understanding of CEFR among educators and the application of a teaching approach on “can do” tasks posed a substantial challenge to teachers. Some however, demonstrated clear opposition to the notion that limited understanding and the “can do” approach represent impediments to the CEFR adoption process. A contrasting trend also emerged regarding the perceived impact of CEFR on teacher autonomy. One of the participants said that the integration of CEFR into the classroom practices would diminish lecturers’ autonomy as educators. However, others remain convinced that their classroom autonomy might not be compromised and they are not daunted by its implementation as they possess a comprehensive understanding of the framework and they can use this comprehension to make better decisions in class. When asked about the training provided for CEFR, the participants agreed that familiarization workshops were conducted to acquaint teachers with CEFR, elucidating its six proficiency levels. They concurred that the series of training and workshops on CEFR were imperative for a comprehensive grasp and assimilation of the framework's concept and utilization. In regard to the familiarization workshops held by the universities, four of the participants agreed that the workshops held significantly enhanced their understanding of CEFR. They agreed that their familiarity and comprehension of the framework improved after the training. As one quoted, “We were exposed to the basic familiarity of the framework which introduces us to the framework and its descriptors”. Meanwhile, two of the participants disagreed with the workshops’ impact. They highlighted the need for more comprehensive CEFR-related workshops to enable effective integration of CEFR descriptors into teaching and learning processes emphasising on the workshops’ insufficiency for classroom implementation. In fact, three of the participants said that they have read particularly the document ‘English language education reform in Malaysia: The roadmap 2015-2025’ and they possess a degree of familiarity with CEFR, facilitated in part by the familiarization workshops they attended. 53

Another challenge pertained to resource-related issues. One participant was quoted saying, “We need more localized contexts for our students as the books we have are from foreign countries”. The absence of sufficient CEFR experts in Malaysia hampers the local production of resources tailored to local contexts. Despite efforts to locate experts, the council has resorted to procuring CEFR-aligned textbooks. However, these textbooks predominantly feature British content, contrasting with the usual practice of localizing content to align with learners’ cultural and contextual backgrounds (Rajabi & Ketabi, 2012). Discussion The participants agreed that the incorporation of CEFR into the education system plays a vital role in aligning with globalization. This suggests that many lecturers perceive the incorporation of CEFR as a necessary step in adapting to the demands of the globalized era. The challenges posed by teachers themselves may hinder the seamless execution of CEFR implementation in Malaysia. This inclination could stem from the notion that teachers might not have been fully prepared to embrace the innovative framework. Their limited comprehension and exposure to CEFR, coupled with the relatively rapid implementation process, could contribute to their reluctance in accepting and effectively integrating the framework into their practices. However, while some participants agreed with this notion, a substantial portion held an opposing viewpoint. They disputed the notion that the incorporation of CEFR within their classroom practices would curtail their autonomy as educators. These participants encompass participants who have a comprehensive grasp of the framework and are not deterred by its presence. Nonetheless, some respondents remain convinced that their classroom autonomy might be affected by the CEFR implementation. Formulating class activities aligned with CEFR descriptors might pose a significant challenge. This phenomenon could stem from teachers’ unfamiliarity with the framework and their lack of adequate training in designing CEFR-aligned activities. However, those who are familiar with CEFR and possess a more comprehensive grasp of the framework are able to effectively craft activities in alignment with CEFR. Some expressed dissent towards the notion that teachers are disinclined to embrace CEFR due to its emphasis on a studentcentred approach, which they believe may not be entirely suitable for Malaysian classrooms. 54

The challenges identified in the interviews encompass challenges stemming from the teachers themselves and challenges related to the practicability of integrating CEFR into their instructional methods. This sentiment likely stems from their limited preparation to fully embrace the novel framework. Their constrained comprehension and limited exposure to CEFR, coupled with the rapid implementation process, could contribute to their reluctance to accept and seamlessly incorporate the framework. “This teacher-related issue might pose challenges to CEFR implementation”, one participant was quoted saying. This may be attributed to the fact that these educators were not adequately equipped to handle the new approach. Consequently, there is a risk that some teachers may possess limited knowledge or even be unaware of CEFR's principles. According to the President of the Malaysian English Language Teaching Association (MELTA), Professor Dr. S. Ganakumaran, certain experts have asserted that the currently utilized imported CEFR-aligned textbooks lack incorporation of local contexts. This deficiency could potentially induce stress among students, particularly those residing in rural and semi-urban areas, as they encounter difficulties in connecting with the content within these textbooks. Additionally, he noted that the imported textbooks, which claim CEFR alignment, do not adequately demonstrate a clear correspondence between their content and CEFR standards (The Star Online, 2018). Despite being endorsed by CEFR experts from Cambridge and procured by the Ministry of Education, local language experts express disagreement with the chosen CEFR-aligned textbooks. This discrepancy of opinions among experts regarding the selected textbook casts doubt on the overall effectiveness of the CEFR implementation process in Malaysia. Limitations, Recommendations and Future Studies This study involved six lecturers, a relatively small representation of university lecturers in Malaysian universities. Thus, future research is advisable to enhance the sample size by including more lecturers. In addition, his study focused on opinions of lecturers from two universities in Selangor exclusively. Conducting a more comprehensive study involving lecturers from other states could yield improved and potentially more accurate findings. It is worth noting that students were excluded from the study. Hence, for a more comprehensive understanding, involving students as primary respondents and conducting classroom observations would be invaluable. These could shed light on the challenges and successes of both teachers and students. Research exploring the implications of using imported materials should also be carried out to ensure their suitability for the Malaysian context. 55

Conclusion In conclusion, teachers embracing CEFR in Malaysia appears to be grounded in a recognition of the potential long-term benefits it holds for students and the overall English language learning landscape. This study sheds light on university lecturers’ perspectives on CEFR implementation in universities in Malaysia. Teachers generally exhibit a positive outlook towards the framework and the council’s roadmap, recognizing the need to enhance English proficiency to compete globally. However, the limited knowledge, minimal exposure, and awareness of CEFR among educators may impede the process. Government officials remain optimistic about CEFR’s potential benefits despite anticipated challenges. Yet, the challenges encompassing educators’ classroom autonomy, resistance, and the lack of CEFR experts, inadequate training, and perceptions of difficulty in integrating CEFR into teaching practices warrant careful consideration. While CEFR adoption in Malaysia is a necessary endeavor, additional time should be allocated to ensure the educators’ comprehensive understanding and readiness. A significant number of lecturers remain unprepared to fully integrate the new framework. Teachers’ resistance and unpreparedness to embrace CEFR can be attributed to psychological factors, particularly feelings of anxiety and nervousness towards novel changes, particularly those impacting educational policies that directly influence their teaching and learning methods. Experiencing anxiety and nervousness when confronted with fresh policies, approaches, or methods is a common occurrence among teachers. However, attributing blame solely to them would be unjust, given the novelty of CEFR and as resistance is a natural reaction. 57

References Darmi, R., Mat Saad, N.S., Abdullah, N., Behak, F.P., Zakaria, Z.A. & Adnan, J. N. I. (2017). Teachers’ views on students’ performance in English language proficiency courses via CEFR descriptors. IJAEDU- International E-Journal of Advances in Education, 3(8), 363–370. http://ijaedu.ocerintjournals.org/download/articlefile/338673 Jerald, G. G., & Shah, P. M. (2019). The impact of CEFR-Aligned Curriculum in the teaching of ESL in Julau District: English Teachers’ Perspectives. In International Journal of Innovative Research and Creative Technology, 4 (6). http://www.ijirct.org/papers/IJIRCT1801023.pdf Lo, Y. Y. (2018). English teachers’ concern on Common European Framework of Reference for languages (CEFR): An application of CBAM. JuKu: Jurnal Kurikulum & Pengajaran Asia Pasifik, 6(1), 46-58. https://juku.um.edu.my/article/view/11174 Mohamad Khair, A. H. & Mohd Shah, P. (2021). ESL Teachers; Perceptions on the implementation of CEFR in Malaysian primary schools: Issues and Challenges. Journal of Advances in Education Research, 6 (1). https://dx.doi.org/10.22606/jaer.2021.61005 Mohamad Uri, N. F. & Abdul Aziz, M. S., 2018. Implementation of CEFR in Malaysia: Teachers’ awareness and the challenges. 3L: The Southeast Asian Journal of English Language Studies, 24(3), 168 – 183. https://doi.org/10.17576/3L-2018-2403-13 Mohamad Uri, N. F. (2021) CEFR in Malaysia: The appropriacy of English syllabus and assessment, stakeholders’ perspectives and challenges. [Doctoral dissertation, Faculty of Social Sciences and Humanities, Universiti Kebangsaan Malaysia, Bangi]. 57

58 Rajabi, S. & Ketabi, S. (2012). Aspects of cultural elements in prominent English textbooks for EFL setting. Theory and Practice in Language Studies Journal, 2(4), 705 – 712. doi:10.4304/tpls.2.4.705-7012 Sabbir, F. (2019). Perceived view of teachers towards Pentaksiran Tingkatan Tiga (PT3) (Form Three Assessment) English Language: A Case Study. Asian Journal of University Education, 15 (3). https://doi.org/10.24191/ajue.v15i3.7819 Sidhu, G. K., Kaur, S., & Chi, L. J. (2018). CEFR-aligned school-based assessment in the Malaysian primary ESL classroom. Indonesian Journal of Applied Linguistics, 8(2), 452-463. https://doi.org/10.17509/ijal.v8i2.13311 The Star Online. (2018, February 4). Let’s have our own textbooks. http://www.thestar.com.my/news/education/2018/02/04/lets-have-our-own-

Designing a Localised CEFR-Based Writing Test Yuziana Yasin , Harshita Aini Haroon Universiti Malaysia Perlis (E-mail: [email protected]) Universiti Malaysia Perlis (E-mail: [email protected]) Abstract This study was conducted as an initiative to design, implement, and score a localised CEFRbased writing test for the admission of international students to a public technical university in Malaysia. Methodologically, this research used qualitative method which was carried out based on the context of institution under study. The main outcome of this study highlights the teachers considerations and suggestions in designing, implementating and scoring a newly developed CEFR-based writing test. As the process involved was conducted within a local instituitional context, addressing the teachers’ considerations allowed anticipated impacts of the English language tests validity to be monitored more effectively. The outcome can further provide input for continuous improvements to the institutional language assessments. Keywords: CEFR-based test, writing, university admission. Introduction The use of Common European Framework of Reference (CEFR) for international students’ admission into higher educational institution can provide a consistent and objective way to assess students' language abilities, regardless of their native language or country of origin. In addition to assessing students’ language proficiency, the use of CEFR is important to track students’ progress in their language learning to ensure that all students receive quality education that prepares them for the challenges of the future. By prioritizing quality education and using language standards like the CEFR, institutions can work towards ensuring that all students have the opportunity to reach their full potential in language learning. At present, international students who intend to study in Malaysia are required to have obtained a score from CEFR aligned tests such as TOEFL, IELTS, and MUET to demonstrate their English language ability at entry point. Despite being able to show that the students have arrived at a certain required level of English proficiency upon their admission to the university, undergraduate students, specifically international students enrolled at the technical university under study previously faced multifaceted challenge in their study that can have several implications for both students and institutions. 60 1 2 1 2

For international students’ admission, it is common that English language assessments from the high stake exam such as IELTS/TOEFL play an important and critical part in the admission to university (Green, 2007). Despite IELTS and TOEFL commonly used for the international students’ admission and linked to the CEFR, these tests only address the proficiency of test-takers across a broad range of proficiency and without reference to any particular context where the descriptors of performance at different levels of the scale are necessarily general (Educational Testing Service, 2019). Therefore, researchers or practitioners using rubrics used in high stake exam like TOEFL rubric to assess student writing cannot communicate students’ performance using CEFR levels (Fleckenstein et al., 2020). In addition, a study on postgraduate classrooms in Malaysia, conducted by (Zainal Abidin et.al, 2015) found a mismatch between standard exam test score and candidate’s actual writing performance. In adapting the CEFR scale, more research need to be carried out (Abu Sufi & Stapa, 2020) and the implementation of CEFR still needs to be improved (Mohd Ali et al., 2018). This is also supported by Zheng et al. (2016) who stated that most of the previous research on the use of the CEFR focused on Europe and it is not so well-known outside Europe. As a matter of fact, there is little discussion on the application of the CEFR in Asian contexts (Zheng et al., 2016). Therefore, this study aims to investigate the challenges faced by the teachers in designing, implementing and rating a CEFR-based test to be used for the international students’ admission at a public higher institution; which would in turn enhance skills and competencies required for the students to succeed in their respective programs. It is hoped that the design of this writing test can strengthen the implementation of CEFR, thus can provide substantial improvements in educational procedures of a language assessment. Literature review In recent years, CEFR has been adopted in many Asian countries as a way to evaluate language proficiency in their own contexts. CEFR has been strongly accepted and vividly seen in the test localisation of Asian nations where countries like Japan and China view the language framework as fundamental to their language reformulation initiatives (Khairul et al., 2021). In aligning CEFR to the local context, the Common European Framework of Reference Japan (CEFR-J) was designed to address the issue that is lacking in Japan to provide a consistent system for measuring learner proficiency and progress in foreign language pedagogy (Runnels, 2013). This tailored version of the Common Europe Framework of Reference (CEFR) was developed to better discriminate incremental differences in proficiency for Japanese learners of English, who tend to fall mostly within the A1 and A2 levels. Additionally, Vietnam also utilised CEFR as a foundation for solid English language education reform in the country. The Vietnamese government has spent a great deal of time to adopt and adapt CEFR to local Vietnamese context. Language policy study on CEFR in the Vietnamese context has also revealed that the utilisation of CEFR as a language framework is seen as the best solution in reforming Vietnam’s English language education (Nguyen & Hamid, 2020). 61

In comparing the local test and larger-scale standardised tests like IELTS and TOEFL, the design of the larger-scale standardised tests must address the proficiency of test-takers across a broad range of proficiency and without reference to any particular context and the descriptions of performance at different levels of the scale are necessarily general (Educational Testing Service, 2019). For example, the scale descriptors in IELTS and TOEFL are associated with a rich research tradition and provide an excellent representation of proficiency across levels of the CEFR. However, general descriptors provide limited information about the finer characteristics of level performance or how a particular score might relate to a test-taker’s performance on critical tasks within an instructional context (Dimova, Yan, & Ginther, 2020). On the other hand, a local test typically clarifies a given level on a scale. Thus, the data that local tests produce can be used to take a closer look at the characteristics of performance within a level that is only generally represented on a largescale test. In Malaysia, a few institutions have conducted attempts to align in-house English Proficiency Test (EPT) to the CEFR scales to gauge the English language proficiency level of its students (Ibrahim et al., 2017). For example, all students (under- and postgraduates) at International Islamic University Malaysia (IIUM) are required to sit for the EPT upon entrance to the university’s academic programmes. In other words, the EPT is a pre-requisite and an assurance that students are ready and language proficient, before the commencement of their studies at their respective faculties. Apart from IIUM, Universiti Malaysia Pahang (UMP) has also undertaken initiatives to align the English language courses and English language proficiency or placement tests (EPTs) to the CEFR (Mohd Ali et al., 2018). One of the reasons of the CEFR aligning at UMP is to ensure that prospective students among international applicants who are unable to meet the English language requirement prior to admissions will have the opportunity to achieve the level of English language proficiency as required by taking pre-sessional English language courses and in-house proficiency or placement tests which can be offered at affordable rates and customised according to the needs of the stakeholders. The contextualisation of the CEFR is thus needed to suit the needs and demands of the stakeholders of Malaysian technical university by identifying the socio educational contexts of the university and align the locally developed test to the CEFR (Mohd Ali et al., 2018). In school, the implementation of CEFR has received mixed reactions especially from teachers who are confronted with a number of challenges. At secondary school level, a study was conducted by Farehah (2017) to investigate English teachers and Ministry of Education officials’ views on the implementation of CEFR in Malaysia. The findings revealed that most of the teachers had very limited knowledge, minimum exposure and low level of awareness about CEFR. Nevertheless, the teachers were optimistic about the idea and believed that the officials in the ministry were also positive about the implementation plan despite the challenges and obstacles perceived. Therefore, the adoption of CEFR in Malaysia is seen as obligatory but sufficient time should be given by the ministry to ensure that all stakeholders are fully prepared and familiar with the framework before it is extensively introduced and used in the country. 62

Therefore, more studies are needed to guide the practitioner on the step-by-step process on the implementation of CEFR-aligned curriculum and assessment in order to carry out the CEFR-aligned process with sufficient determination and thoroughness. Given the increasing interest in applying the CEFR outside of Europe, the process of developing alternate versions “to suit local needs and yet still relate back to a common system” (Council of Europe, 2001, p. 32) requires further study. O’ Sullivan’s validity model for localisation The process of test localisation, or adapting a test to a specific cultural and linguistic context, is crucial in ensuring that the results of the test accurately reflect the abilities of the test taker. One framework for validating a test for localisation is O’Sullivan’s (2011) model, which includes three basic elements: the test taker, the test system, and the scoring system. In the context of test localisation for the CEFR writing assessment, test taker characteristic of O’Sullivan’s (2011) includes the consideration of the cultural and linguistic background of the test taker. The second element of O’Sullivan’s (2011) model is the test system which this includes the format of the test, the task types, and the scoring criteria. The format of the test should be appropriate for the test taker, taking into account their cultural and linguistic background. Additionally, the task types and scoring criteria should be appropriate for the level of proficiency being assessed. Lastly, the third element of O’Sullivan’s (2011) model is the scoring system which includes the use of trained raters to score the test and the use of appropriate scoring criteria. The use of trained raters is important to ensure that the test is scored consistently and accurately, regardless of the cultural and linguistic background of the test taker. Additionally, the use of appropriate scoring criteria is important to ensure that the test accurately measures the level of proficiency being assessed. The three basic elements of the model – the test taker, the test system, and the scoring system – are all crucial to ensuring that the test is appropriate for test takers from different cultures and linguistic backgrounds, and that the test accurately measures the level of proficiency being assessed. Methods Participants Eight English teachers from a public institution participated in this study. A focus group discussion (FGD) was administered with the participants to provide their perceptions after undergone the process of designing, implementing and rating a localised CEFR-based writing test for the international students’ admission. Participants were selected based on the criteria that they had sufficient experience with and knowledge of the kind of writing tasks and language assessment normally encountered in the undergraduate academic context, the level of English writing proficiency required to function properly in undergraduate study, and the institution writing test format. 63

Context The university under study mainly offers various engineering programmes as well as a small number of business and entrepreneurial programmes. Most of the international students enrolled at this university are engineering students. Hence, the design of the entrance test has taken into consideration of the population and context, in which the test can be used. As most of the academic programmes offered at university under study are delivered in English, this study focuses on the English language entrance test. Instrument The participants were prompted with a few general questions related to the issues that have been taken into account in designing, implementing and rating a localised CEFR-based writing test. Participants were required to provide their feedback based on the following questions: What are the consideration that you have been taken in designing, implementing and scoring a localised-CEFR based writing test for the international students’ admission? 1. 2.What suggestions do you have to improve the process? Procedure Participants were involved with either the process of designing, implementing or rating a localised CEFR-based test to be used for the international students’ admission at the higher institution under study. After completing the process, the participants were invited for a focus group discussion. They were required to share the consideration that they have taken into account throughout their participation of the process in designing, implementing or rating a localised CEFR-based test to be used for the international students’ admission. Their feedback were recorded and grouped according to the O’Sullivan’s (2011) model. Findings/Results The feedback recorded from the FGD were presented according to the following categories based on O’Sullivan (2011) localisation model: a) The test taker: The participants pointed out that they have considered the test takers’ cultural and linguistic background in designing the writing test. One of the participants highlighted on the need to design writing tasks that are familiar to the students’ background: Student 1 The student chose the topic of mental health for her presentation which is a complex subject suitable for C1 level. In addition, we found that she enhanced the audience’s understanding of the concept of mental health by citing an authoritative figure in the field in defining mental health. This can be illustrated below: 64

“ When we designed our question, we designed two tasks because our argument is for the students to be able to interpret nonlinear data (C1) and then also to an argument essay as writing task (B2). We can come up with question that are familiar to the international students” During the selection of the suitable English course to be adapted for the entrance test, participants pointed out that they have considered the students’ language ability and experience. The participants highlighted that: “The potential international students will be able to write their view if the essays are related to their personal life”.(Teacher 1) “The essay topic that are familiar to the students can help the students to express their ideas in English”.(Teacher 2) Apart from that, when implementing the test to the students, one of the participants pointed out that it is important to ensure that the instruction is clear to the students. He stated that: “I explained to the students the instruction before they started writing the essay because one of the students asked me: “Sir do we write more than 200 words, or do we write just the exact amount or can we write less”. So, I answered to them, for Part A because it is reflective writing, based on my understanding it is at B1 level, they cannot write more than 200, because ‘write the topic in 200 words’, that’s mean, they can either write 190 something but cannot be 200 words.”. (Teacher 6) Participant also pointed out that some test takers who did not have a strong command of the language preferred to write fewer than the word limit, hence the selection of the essay topic must consider the students’ background knowledge: “I know my student’s ability, they are worried that they might not be able to write more than 200 words. They are worry about the essay topic before they start their writing. I need to explain to them on the topic and asked them to write based on their personal experience. Then they feel more confident to write about it.”(Teacher 2) a) The test system: The second element of O’Sullivan’s (2011) model, the test system was discussed by the partipants in designing a localised CEFR-based writing test for the international students’ admission. One of the participants pointed out that benchmarking the test with the international test and also from standard entrance test used in Malaysia can help the teacher in designing the test that suitable to the international students based on the expected CEFR level: “We can also adapt from IELTS, or Cambridge placement test that has been aligned with CEFR.. We can also looked at MUET. MUET also endorse. So we can also take a bit and pieces from MUET. From Cambridge Placement test.” (Teacher 2) 65

On the other hand, another participant suggested to use the current English language assessment in the designing a CEFR-based test for the institution entrance test: “We developed our test by looking at the existing assessments that we're using in our respective universities. Like what I did, our courses has been aligned with CEFR, so what I did was, we selected one of the courses. We looked at the existing test, that we actually do test our students here and based on the rubric, we designed for our assessment. Based on our existing language courses, we can consider the writing tasks that are suitable for international students. ” (Teacher 3) Apart from that, the teachers highlighted on the consideration of suitability of the time allocated for the writing tasks based on the CEFR level. In designing the test, one of the participants suggested to allocate more time to the higher level CEFR writing tasks to ensure low proficiency students were given ample time to write their response: “If the test is targeted for B1 level, I think the time limit should be increased. For example, when we set summary test for our students, they need to write 150 words, but we give them more than 30 minutes. So now the words limit is not more than 200 words, so I think it should be increased a little bit. Since the task is B1, the lower proficiency level students might struggle more. Especially my international students, they may need more time to write in English. So, they need more time”. (Teacher 4) Further discussion was also given to the word limit and time allocation when the participants raised the issue on the expected number of words for the students to provide their response. To gauge students’ language proficiency, one of the teacher suggested to increase the word limit a bit so that students can express more thought and view on the given topic: To comment on Teacher 5’s concern on the test takers performance, Teacher 3 pointed out that Task A was set to be in 150 words earlier due to the expectation of the writing task to measure students’ ability at B1 level. However, Teacher 3 supported the view to increase the word limit as to provide more opportunity to the students to express their language proficiency through their thought and view on the given topic: “We have taken sample from one of our courses, which is English 1and English 2.. For English 1, international students come with A2, they exit with B1. So that’s why their level is very basic. English 1 students that come to our university, if they don’t meet the attainment of Band 3 for MUET, they get Band 2 or Band 1, they have to go English 1 first before they enrol into the mainstream courses..or do we want to upgrade it? The level is not too high right? We are targeting B1, because our respondents are international students, isn’t it?” “The first question just now is at B1 level, is 150 words sufficient to gauge the level of the students? 150 words is like one paragraph, like 120-150 words. Maybe we can have at least 200 words. (Teacher 5) Participants hence in the same view to raise the word limit and believed that slight increase to the word limit is sufficient to measure students’ language proficiency. The teachers believed that “for B1 level, 200 words essay is Ok”. Moreover, one of the teachers suggested that the instruction need to be clearer to the international students as some of them are low proficiency students with A1/A2 level: 66

“The instruction is too long and looks complicated to the students... the students may feel worried to write their view when there are too many things to understand from the instruction. For example, the instruction required the students to do short introduction to the situation, evaluate the most important things about the experience, including solutions to problems, write what you would do differently next time and write what you learned overall. There are too many words in the instructions. Students will have problem to understand... It can cause frustration to the students”. (Teacher 7) As for improvement, the participants suggested the instruction to be “simple and straightforward”. c. The scoring system The third element of O’Sullivan’s (2011) model, the scoring system was also taken into account by the participants in scoring a newly developed CEFR-based writing test. Teacher 5 highlighted on the effectiveness of the rubric in guiding the raters to examine the test taker’s writing performance. She stated that “I was quite uncertain of myself, not confident with myself when I'm giving the marks”. Teacher 6 also pointed out that she faced difficulty in using the rubric when she said “When I marked, the rubric is a bit unclear, especially if we can see that if it is C1... it says ‘without having to restrict what he/she want to say’, that’s a bit weird. The other one is A2, language part says, grammar but it is on grammar accuracy, it says that ‘use simple structure correctly’. But the second part still systematically did basic mistake.” Apart from the readability of the rubric, Teacher 7 addressed the issue on the marks allocated in the criteria of the rubric. She pointed out that: “I think when I marked that day, let say that for part A, the student got C1, and then Part B the student got B2, and then the total result that the student get is like C1. So, I think the arrangement probably need to double check on the marks given”. When scoring the esssay, Teacher 8 felt that test takers’ performance in the test were in a good range. Teacher 8 mentioned that “When marking the test paper, I realized that the student’s level was quite good”. Despite low proficiency students preferred to write fewer than the word limit due to insufficient vocabulary to expand their ideas well, high proficiency students were able to perform well in their writing. The good performance of the students’ score has also sparked curiosity to Teacher 4 on the cheating when she said, “Is this test conducted online or face to face?” Teacher 4 concerned on students’ possibility of cheating was clarified by Teacher 1when she said that “if you already wondering whether it is online or face to face, I was there as well during face to face, some students are proficient. Most of them were ex-IEC students”. As the test was administered in a control environment with test invigilators, Teacher 4 believed that “the test session has strengthened the reliability of the test taker’s performance”. 67

Discussion Overall, the teachers’ consideration in designing, implementing and scoring a localised CEFR-based writing test for the international students’ admission were matches with O’ Sullivan’s validity model for localisation. Moreover, most consideration were connected to all the three elements of O’Sullivan (2011) validity model and contextualisation of CEFR to the local context to suit the needs and demands of the stakeholders of Malaysian technical university. First, in term of the element of test taker , the students’ question to the teacher to get further clarification on the word limit and essay instruction suggests that in a localised test, the test takers relied heavily on the test administrator for guidance, clarification, or assistance. Here, the considerations given to the test takers' cognitive and experience can greatly impact test results and overall validity. Hence, the feedback provided by the test administrator indicates that test administrator’s involvement influenced the test takers’ experience which potentially affected their test results or outcomes. Apart from that, the feedback given by the teacher highlights the importance of the consideration of test system in O’Sullivan’s (2011) model which specify that in the context of the CEFR writing assessment, test takers plan their writing timing to prepare for their writing tasks in order to meet the expectation of score weighting. Hence, the knowledge of how their performance will be scored is crucial and thus contributed to their better performance. On the other hand, the physical/physiological characteristic of the test taker was not mentioned in the discussion. This suggests that test taker physical/ physiological characteristics (age, gender, short-term ailments and longer-term disabilities) seem to have almost no place in the CEFR linking process at the specification session as the focus during this session was more on the test taker’s cognitive and experiences. Secondly, in term of test system (O’ Sullivan, 2011), despite some of the teacher suggested to increase the time spend for the writing tasks, the teacher that have invigilated the test session a pointed out that they did not receive any request from the students to extend the time. Hence, a clear guideline on the time suggested to be allocated for the writing tasks as specified in the test question were able to guide students to prepare for their writing within the word limit that have been set. In addition, the participants also highlighted on the consideration the test format and instruction in designing a CEFR-based writing test. The process of benchmarking the test with the other standardise English tests and university English curriculum allowed the teacher to design suitable writing test format for the newly developed entrance test. As an essay task that is targeted to be at B1 level is also expected to be taken by the lower-level proficiency students with A1/A2 level, therefore the writing test format should be clear and simple. Lastly, in term of scoring system, the teachers feedback suggests that it was challenging to use the rubric when they were required to mark the essay individually without rater’s training. Hence, for the improvement, the rubric need to be revised so that it is more readable and clearer to the teachers in guiding them to assess the students’ performance. Moreover, the rater’s feedback on the cutoff score suggests that the raters also faced difficulty with the calculation of the overall marks to be mapped to the CEFR. Despite the participants are experienced English teachers and familiar with the CEFR, a raters’ training for marking standardization is needed to ensure that all the raters were able to use and apply the descriptor in the rubric well and consistently. 68

Nevertheless, the results suggest that replications of the current study in other institutional context might be useful in order to determine whether refinement or modification of the test questions and rubric are required. The attempts to design, implement and scoring a localised CEFR-based writing test for the international students’ admission in this study do not guaranteed to provide the same challenges and solution for every group of test takers or teachers. In the current study, the consideration were not consistently taken by all the participants, which has implications for the localised CEFR-based users in the institution under study. Hence, changes and adaptation of CEFR are needed based on instituitional context. Limitations, Recommendations and Future Studies This study has its own limitation whereby the process of designing, implementing and scoring of a CEFR-based writing test only focuses on the undergraduate students for the international students’ admission. Secondly, the numbers of teachers and students involved in this study do not involve the whole population of English teachers and undergraduate international students at the university under study. Hence, the data might not represent the voice of the whole population. The reason why the researchers select only 8 students and 30 international students as participants to this study is because of the time constraint and the limitation of international students available to participate in this study. As a result, this research has been narrowed down to the numbers of respondents available and at the same time was enough for the data analysis. For the future study, it is recommended that the area of research can also be expanded to the other language skills such as reading, speaking, and listening skills. Specifically, the study should focus more on the development of the CEFR-based test to be used for the admission of the international students to the university that cover all the four skills which could bring to a more comprehensive analysis on the considerations that have been taken into account in designing, implementing and scoring a localised CEFR-based entrance test to measure international students’ language proficiency. Future researchers are also suggested to discover on the development of the CEFR-based entrance test as part of the admission requirement for the international students to the universities. By doing this, the researcher might reveal the English proficiency of the potential international students. At the same time, future study may provide useful information as a solution to tackle the lack of proficiency among international students in their tertiary levels of education in Malaysia. Conclusion The current study intended to explore the teachers’ perspective in the attempt of designing, implementing and scoring a CEFR-based writing test for the international students’ admission to undergraduate programs at a local institution in Malaysia. Ultimately, the results suggest that the process of contextualization of a generalized European framework for local purposes outside of Europe is feasible and was relatively successful. However, training and standardisation process among the teachers are suggested in helping the teachers overcoming their challenges. Therefore, additional support is still required so that the results obtained from the test can be used in the construction of curricula, materials and assessments for improving language learning among international students in the tertiary institutions in Malaysia or as a model for any organization looking to localise a general framework of reference. 69

References Abu Sufi, Mohd. K., & Stapa, M. (2020). Should the CEFR Illustrative Scales be localised to Malaysian Higher Education Standards? – A Conceptual Paper. International Journal of Academic Research in Business and Social Sciences, 10(9). https://doi.org/10.6007/ijarbss/v10-i9/7882 Educational Testing Service. (2019). Performance Descriptors for the TOEFL iBT ® Test. Fleckenstein, J., Keller, S., Krüger, M., Tannenbaum, R. J., & Köller, O. (2020). Linking TOEFL iBT® writing rubrics to CEFR levels: Cut scores and validity evidence from a standard setting study. Assessing Writing, 43(July 2018), 100420. https://doi.org/10.1016/j.asw.2019.100420 Green, A. (2007). IELTS Washback in Context: Preparation for Academic Writing in Higher Education. Studies in Language Testing, 386. Khairul, M., Sufi, A., & Idrus, F. (2021). A Preliminary Study on Localising the CEFR Written Production Descriptor to Malaysian Higher Education Context. 3(2), 1–15. Mohd Ali, Z., Ali, F., Mohd Radzuan, N. R., Nik Mohd Alwi, N. A., Abu Kassim, N. L., & Mohd Don, Z. (2018). Contextualising the Cefr: the Universiti Malaysia Pahang English Language Proficiency Writing Test. ICERI2018 Proceedings, 1(April 2019), 4892–4902. https://doi.org/10.21125/iceri.2018.2114 Mohd Sallehhudin Abd Aziz, N. F. M. U. (2017). CEFR in Malaysia: Current issues and challenges in the implementation of the framework. International Journal of Academic Research in Business and Social Sciences. O’ Sullivan, B. (2011). Language Testing. In The Routledge handbook of applied linguistic (pp. 259–273). https://doi.org/10.1016/s0346-251x(0)00022-2 Zainal Abidin, S. A., & Jamil, A. (2015). Toward an English Proficiency Test for Postgraduates in Malaysia. SAGE Open, 5(3). https://doi.org/10.1177/2158244015597725 Zheng, Y., Zhang, Y., & Yan, Y. (2016). LITERATURE REVIEW_a case study outside Europe.pdf. 70

Unveiling Tertiary Level Students’ Confidence in English Oral Comprehension: Insights from a CEFR-Informed Course Mohd. Khairul bin Abu Sufi , Azura Abdul Razak English Language Division, Centre for Languages and Pre-University Academic Development (CELPAD), International Islamic University Malaysia, Gombak (E-mail: [email protected]) English Language Division, Centre for Languages and Pre-University Academic Development (CELPAD), International Islamic University Malaysia, Gombak (E-mail: [email protected]) Abstract This study investigates the self-assessment of speaking and listening skills among Common European Framework of Reference (CEFR) students. The aim of the research is to explore how students perceive their own language proficiency and identify potential areas of improvement. The study specifically focuses on the students’ perceived confidence in communication and the extent to which they believe their fluency in the target language has improved. A simple 6 question questionnaire was employed to collect data from a diverse sample of students studying English in a university’s pre-sessional programme. Selfassessment questionnaires were administered to gauge the students’ perceptions of their oral comprehension skills. The findings of this study revealed that students reported confidence in their oral comprehension abilities in the target language. This suggests that the instructional strategies employed in the CEFR curriculum have effectively boosted students’ self-assurance in their oral comprehension abilities. However, despite the perceived increase in confidence, the students indicated that their overall fluency in the language did not improve significantly. This study serves as a foundation for further research to explore effective instructional strategies that bridge the gap between perceived confidence and actual fluency, ensuring a well-rounded language learning experience for CEFR students. Keywords: CEFR, listening, speaking, higher education, self-assessment Introduction The Common European Framework of Reference for Languages (CEFR) is a widely recognized framework developed by the Council of Europe to assess language proficiency. It categorizes language competency into six levels, from A1 to C2, and covers speaking, reading, listening, and writing skills. The CEFR’s descriptors provide detailed explanations of what learners can achieve at each level, aiding teachers and learners in setting goals, creating resources, and monitoring progress. The framework is regularly updated to meet the evolving needs of language learners and instructors worldwide. 72 1 1 2 2

CEFR in Malaysia In Malaysian universities, the Common European Framework of Reference for Languages (CEFR) has become a crucial part of language education. Ismail et al. (2017) discuss the various roles of CEFR, emphasizing that it goes beyond being just an assessment tool. Instead, it serves as the foundation for evaluating language proficiency, guiding educators as they navigate the complexities of linguistic development. Additionally, CEFR plays a key role in assessing language proficiency in Malaysia, providing a standardized measure for linguistic abilities, ensuring fairness, and promoting a shared understanding of proficiency levels across different academic contexts. Furthermore, CEFR’s influence extends to curriculum design and development. Ismail et al. (2017) describe how the framework acts like a map, guiding the development of language programmes. It’s not a rigid set of rules but a flexible guide, allowing educators to tailor programs to meet the unique needs of their students while adhering to global standards set by CEFR. Another significant impact is the standardization of language proficiency settings in Malaysian education. Ismail et al. (2017) highlight how CEFR acts as a unifying force, fostering a common language among educators, students, and institutions. This commonality not only creates a cohesive educational environment but also facilitates smooth transitions for students within the Malaysian academic landscape. In conclusion, CEFR deeply influences the educational landscape in Malaysia, not only shaping how language proficiency is assessed but also impacting the core of language education. As demonstrated by Ismail et al. (2017), CEFR actively participates in shaping the narrative of linguistic development in Malaysian universities. CEFR IN IIUM In the context of IIUM, students need to score a minimum of band 6 in the English Placement Test, which all students must sit for upon entering the university, in order to enter their respective areas of studies. Failure to attain a band 6 will result in students to undergo pre-sessional English classes. Currently, there are seven levels of English offered in IIUM’s Centre for Languages and Pre-Academic Development (CELPAD), where levels 1-3 cater to beginners, levels 4-5 for intermediate and level 6 for upper intermediate users of English. 73

Additionally, a remedial class caters for students who have been unable to achieve a band 6 in their EPT after three consecutive tries. Assessment wise, the CEFR has been adopted by CELPAD as a reference point in designing their English courses, as well as in the design of EPT paper and in the assessment of the EPT. The EPT descriptor is in line with the CEFR descriptor, which benchmarks the students’ abilities in writing, reading, listening and speaking skills. Statement of the Problem The focus of the CEFR is mainly on the educational aspect of language and does not place emphasis on the aspects of neither sociocultural nor pragmatic aspects of communication. Many language experts believe that communication is more in depth than just grammatical competencies. Real life communication skills are often overlooked when dealing with assessing a student’s ability in a certain language as assessments are based on unrealistic scenarios and settings. Due to this, it could be said that what students perceive about their language skills may not reflect their actual proficiency in the language. Data pertaining to students’ perception of their language skills are needed so that researchers can be clear of how students use the CEFR framework in their daily use of the language. This is to ensure that students will have a realistic understanding of their language skills. There are many aspects that must be taken into account regarding how students perceive their language abilities, and this includes their cultural background, knowledge and exposure to language that they aim to master; their educational background and their level of self-competence are also other factors that must be considered. An investigation on these factors is needed to uncover a deeper insight on how students assess their language proficiency. The data obtained will lead to instructors to develop various strategies on improving their students’ learning outcome. Besides that, obtaining information on the students’ perceived language abilities, be it positive or negative, will assist academics to establish areas of language learning that need to be improved. The students’ feedback will provide an insight on which areas of language learning that the students most need assistance in. From this, academics will be able to develop better pedagogical access and methods to improve on the areas of language learning that need to be developed further. 74

Research questions The research study aimed to address the following key research questions: RQ1: What are students’ perceptions of their English language speaking skills upon completion of a CEFR-aligned course? RQ2: What were the difficulties encountered by students throughout the CEFR oral comprehension course? Literature review CEFR assessment grid The CEFR has garnered considerable recognition as a comprehensive pedagogical framework for language instruction, learning, and assessment (Council of Europe, 2020). It offers a standardized and transparent system for delineating language proficiency levels and learning outcomes, thereby facilitating effective communication and collaboration within the domain of language education (North, 2017). An invaluable tool within the CEFR is its self-assessment Grid, enabling learners to appraise their language proficiency across diverse levels (Council of Europe, 2020). The selfassessment Grid has emerged as a pragmatic and learner-centric approach to self-evaluation, affording learners the means to monitor their progress and establish attainable learning objectives (Beacco et al., 2016). The CEFR is rooted in a robust theoretical foundation encompassing pivotal constructs such as language proficiency, communicative competence, and language learning progression (Council of Europe, 2020). It embraces a holistic perspective on language acquisition, placing emphasis on both communicative and intercultural competences (Little, 2018). The self-assessment Grid embedded within the CEFR is meticulously designed to reflect these theoretical underpinnings, endowing learners with a structured framework to self-appraise their individual language proficiency in listening, speaking, reading, and writing (Council of Europe, 2020). By aligning self-assessment practices with the CEFR descriptors, learners can attain enhanced comprehension of their strengths and areas requiring further development across different language skills (Beacco et al., 2016). However, despite its utility as a self-assessment instrument, the effectiveness of the CEFR self-assessment Grid can be subject to a multitude of factors. Learner-centric factors, 75

including motivation, language background, and self-perception, wield considerable influence over the accuracy of self-assessment outcomes (Fulcher, 2012). Notably, learners displaying heightened motivation and possessing favourable self-perceptions of their linguistic prowess may exhibit a tendency to overestimate their proficiency, while those harbouring reduced motivation or possessing limited exposure to the target language may exhibit an inclination to underestimate their capabilities (Lee, 2012). Instructional factors similarly exert an impact on the quality of self-assessment practices. Teacher guidance and training in the utilization of the self-assessment Grid assume pivotal roles in fostering learners’ comprehension of the descriptors and their apt application (Beaven & Sadler, 2016). Educators are uniquely positioned to provide valuable feedback and scaffold learners’ self-assessment endeavors, thereby augmenting their ability to accurately gauge their language proficiency (Evans & Green, 2019). Furthermore, cultural factors can engender variances in learners’ perceptions and evaluations of their language proficiency. Distinct cultural backgrounds and educational systems may engender divergent beliefs and expectations concerning language acquisition, consequently yielding dissimilarities in self-assessment outcomes (Taylor, 2012). Cultural biases and sociocultural influences can impinge upon learners’ interpretations of the selfassessment Grid, thereby influencing their evaluative judgments of their linguistic aptitude (Elder et al., 2019). Notwithstanding its merits as a means of self-appraisal, it is essential to acknowledge the challenges and limitations associated with the CEFR self-assessment Grid. Subjectivity and biases inherent to learners can introduce inaccuracies and discrepancies in the resultant evaluations (Milton, 2011). Moreover, the efficacy of the self-assessment Grid is contingent upon learners’ capacity to objectively evaluate their own performance, an aspect susceptible to the influence of factors such as anxiety, deficiencies in metacognitive skills, or limited exposure to diverse linguistic contexts (Shepard, 2016). Listening skills Listening skills are essential in various aspects of life, including educational settings. In higher education, students are expected to engage in active listening to comprehend lectures, discussions, and presentations. Therefore, understanding the significance of developing effective listening skills is essential. This literature review aims to explore existing research on listening skills for higher education students, examining theoretical frameworks, benefits, strategies, challenges, and implications for future research and practice. 76

Listening skills encompass the ability to process and comprehend oral information. Theoretical frameworks describe listening as a multifaceted process involving cognitive and affective components. Information processing models shed light on the stages of listening comprehension, highlighting the importance of attention, perception, interpretation, and memory (Li & Wang, 2019; Rahmawati et al., 2017). Moreover, the link between listening comprehension and academic performance has been established, emphasizing the need for proficient listening skills in higher education. Developing effective listening skills brings numerous advantages to higher education students. Firstly, enhanced learning and academic achievement occur when students can accurately understand and retain information presented in lectures and discussions (Milton, 2011). Active listening also fosters improved communication and interpersonal skills, promoting effective collaboration and empathy. Furthermore, listening skills contribute to critical thinking and problem-solving abilities, as students gain a deeper understanding of complex concepts and diverse perspectives. Additionally, proficient listening facilitates language acquisition and multicultural understanding, preparing students for professional and workplace contexts (Milton, 2011). To cultivate listening skills in higher education, various strategies and interventions can be employed. Pre-listening activities, such as activating prior knowledge and setting listening goals, help students focus their attention (Madani & Kheirzadeh, 2022). Active listening techniques, such as maintaining eye contact, paraphrasing, and asking clarifying questions, enhance comprehension and engagement. Note-taking strategies, such as structured outlines or visual mapping, aid in information retention. Metacognitive strategies, such as selfassessment and reflection, promote self-regulation and awareness of listening processes. Leveraging technology-assisted listening tools and resources can also facilitate skill development. Finally, collaborative learning and peer-to-peer interactions encourage active engagement and foster listening skills through discussion and feedback (Harney et al., 2017). Despite the benefits and strategies available, challenges exist in developing listening skills among higher education students. Multitasking and digital distractions pose significant obstacles to effective listening, as students are often tempted to divert their attention to electronic devices. Language barriers and cultural differences can impede comprehension for international students or in diverse classrooms (Park, 2022). Listening anxiety and lack of motivation may hinder students’ willingness to engage actively. Assessment and feedback 77

limitations also pose challenges, as traditional evaluation methods may not capture the full range of listening abilities. Moreover, integrating listening instruction across different disciplines can be challenging due to varying instructional approaches and priorities (Borhany et al., 2015). Methodology This study employed a simple mixed methods approach to gather information from students regarding their English language proficiency after successfully completing a speaking and listening course aligned with the Common European Framework of Reference (CEFR) at a public higher education institution in Malaysia. Prior to commencing the study, permission was obtained from the Director of the university’s language centre. The participants of the study consisted of 65 students who were placed in Level 6 of the university’s pre-sessional programme. Upon analysing the demographics, it was observed that the majority of participants were female students, accounting for 77% of the sample, while the remaining participants were male students. Furthermore, it was noted that the majority of participants were from the Kulliyah of Economics and Management Sciences, followed by the Kulliyyah of Nursing and IRKHS, which constituted approximately 10-13% of the study population. An examination of the demographics also revealed that a significant proportion of the participants were from the Eastern states of Malaysia, namely Kelantan, Pahang, and Terengganu, accounting for nearly half of the entire study population. The tables provided below offer a concise summary of the reported demographics. 78

79 Chart Kulliyyah Chart background (state)

80 Data Collection Instrument Data for this study was collected through the utilization of a questionnaire that was developed by adapting the self-assessment grid found in the published CEFR manual (Council of Europe, 2020, p. 177). The questionnaire was specifically designed to assess the participants’ proficiency in the Oral comprehension skill of the CEFR Reception mode. To create the questionnaire, the researchers extracted the can do statements corresponding to all six levels of the CEFR self-assessment scale for the Oral comprehension skill (refer to Appendix A for details). For instance, the can do statement for level A1, “I can recognise familiar words and very basic phrases concerning myself, my family and immediate concrete surroundings when people speak/sign slowly and clearly” was reproduced in the questionnaire. To quantify the participants’ self-assessment, a five-point Likert scale was added, ranging from 1 (strongly disagree) to 5 (strongly agree). Although the CEFR can do statements were used verbatim, the word “sign” however, was omitted as it was not relevant to the context of the participants in this particular study. Subsequently, the students were instructed to assess their own Oral comprehension skill based on their perceptions after successfully completing a 14-week English language course at the university. In addition to the 6 statements that investigated their oral comprehension skills, two additional open-ended questions were added to elicit information on obstacles that the students faced throughout the semester. The additional questions are as follows: Q1. Which aspect(s) of the teaching and/or learning could be improved? Please explain briefly. (e.g., better textbooks, more hours for listening or speaking classes etc.) Q2. What is your opinion about your English language abilities (listening and speaking) after completing your English language classes in CELPAD? In order to simplify the data collection method, we digitized the self-assessment questionnaire using Google Forms. The creation of an electronic questionnaire would ease the distribution of the questionnaire, and at the same time, data would be collected and tabulated automatically. After having digitized the CEFR self-assessment questionnaire, we distributed the questionnaire link to each instructor teaching at the L6 level. This enabled them to conveniently share the questionnaire with their respective students.

81 M SD A1 4.35 0.89 A2 4.15 0.79 B1 4.14 0.70 B2 3.91 0.85 C1 3.80 0.84 C2 3.63 0.87 The questionnaire used in the study was designed by incorporating the original can do statements of each CEFR level for the Oral comprehension skill. Due to this approach, it was deemed unnecessary to pilot the instrument. However, in contrast, the open-ended questions utilized in the study were thoughtfully crafted, taking into account the insights derived from an extensive review of relevant literature. These open-ended questions were then piloted with a separate class that was not involved in the main study. This piloting phase aimed to test the clarity and effectiveness of the open-ended questions before their inclusion in the actual research. Findings Research Question 1 Research Question 1 of the study aimed to investigate the perceptions of students regarding their oral comprehension skills in the English language. This investigation specifically focused on students who had enrolled in and successfully completed a course informed by the CEFR at a public university in Malaysia. Table 1 below shows the tabulated mean and standard deviation values in correspondent to the CEFR proficiency levels: Results tabulated from the self-assessment scale revealed that the mean score decreased with the increase of the CEFR proficiency levels. This indicated that students recognised their level of proficiency based on the CEFR self-assessment scale as a mean score of >4 was recorded for levels A1, A2 and B1. Interestingly, a mean score of <4 was recorded for levels B2, C1 and C2 indicating

82 that lesser students perceived themselves as possessing the proficiency level of the higher CEFR proficiency scale. The data presented in the aforementioned table demonstrates a descending trend in the mean scores of students’ self-assessment, indicating a consistent alignment with the established difficulty levels defined by the Common European Framework of Reference (CEFR). This substantiates the notion that the self-assessment descriptors outlined in the CEFR can be effectively utilized as a valuable tool for individuals seeking to evaluate their language proficiency levels. The subsequent sections of this paper will proffer an analysis of the findings based on the individual questions. Q1: I can recognise familiar words and very basic phrases concerning myself, my family and immediate concrete surroundings when people speak slowly and clearly. The findings from this question revealed that over 80% of the respondents held the perception of being capable of recognizing familiar words and very basic phrases in the English language. The majority of participants either agreed or strongly agreed with this statement, indicating their confidence in possessing such skills. However, a portion of the respondents exhibited either indifference or disagreement regarding their ability to achieve this level of proficiency. Such perceived inability was unexpected considering that the statement reflects the expected abilities of CEFR A1-level learners, while the participants were enrolled in a B2-level course. The table below provides information of the finding:

83 Table 1: The mapping of Learning Outcome (LO) against Can do statements for giving presentation CEFR C1 level Q2: I can understand phrases and the highest frequency vocabulary related to areas of most immediate personal relevance (e.g. very basic personal and family information, shopping, local geography, employment). I can catch the main point in short, clear, simple messages and announcements. In relation to question 2, the results indicate that slightly over 80% of the respondents expressed their perception of possessing the oral comprehension abilities typically associated with learners who have reached the A2 level. This finding aligns with the pattern observed in the previous question. However, it is noteworthy that a higher proportion of students (10%) displayed indifference towards their abilities to achieve such proficiency, while a small fraction (3%) explicitly stated their inability to attain the oral comprehension skills expected of A2-level learners. The tabulated data presented in the table below provides a clear representation of these findings: Q3: I can understand the main points of clear standard language on familiar matters regularly encountered in work, school, leisure, etc. I can understand the main point of many radio or TV programmes on current affairs or topics of personal or professional interest when the delivery is relatively slow and clear. Question 3 of the questionnaire sheds light on the oral comprehension abilities of B1-level learners. In relation to this particular question, a notable majority of participants expressed agreement with possessing the skills expected to be acquired at the B1 level of the CEFR. However, despite the higher rate of agreement, approximately 10% of the respondents either displayed indifference or disagreed with their ability to attain the oral comprehension skills

84 outlined by the CEFR at the B1 level. Q4: I can understand extended talk and lectures and follow even complex lines of argument provided the topic is reasonably familiar. I can understand most TV news and current affairs programmes. I can understand the majority of films in standard language. This specific question serves to assess the proficiency of B2-level learners in terms of their oral comprehension. Given that the participants in the survey were all engaged in English studies at a B2 level, we expected a considerable level of agreement with the statement. Upon analysing the collected data, our expectations were confirmed, as just under 90% of the respondents either agreed or strongly agreed with possessing the abilities typically associated with B2 learners. In contrast, only 10% expressed uncertainty or disagreement regarding their perception of themselves as possessing the prescribed abilities. The following table illustrates the data:

85 Q5: I can understand extended talk even when it is not clearly structured and when relationships are only implied and not signalled explicitly. I can understand television programmes and films without too much effort. As Q5 illustrates the abilities of learners at level C1 of the CEFR, we did not expect many responses for the categories “agree” and “strongly agree” because the abilities outlined at this level surpasses the participants’ B2 abilities. To much of our surprise, a whopping 66% of participants agreed that their oral comprehension skills were at level C1 - 1 band higher than what they were taught in the curriculum. In addition, 30% of the participants indicated that they were unsure whether their oral comprehension abilities matched those of C1 learners while around 5% stated that they were not at this level.

88 Q6: I have no difficulty in understanding any kind of oral language, whether live or broadcast, even when delivered at fast natural speed, provided I have some time to get familiar with the accent. Q6 represents the highest level of abilities based on the CEFR scale. Again, we would not expect many students to rate themselves as being at this level since the classes were targeted at qualifying students as becoming B2 proficient. However, slightly more than half of the students who participated in this study indicated that they recognised themselves as being C2 proficient when it comes to their oral comprehension skills. Although the CEFR indicated that students would only be able to comprehend “oral language” that is “fast natural speed” “provided” that they were able “to get familiar with the accent”, the curriculum that was introduced to the students exposed students to various English language accents and oral reception texts at speeds that were either native or non-native. Therefore, this could become the justification for the ‘more than expected’ responses to the C1 statement of the oral comprehension skill as students at Level 6 were already exposed to different receptive texts that were either native or non-native in the classroom. Research question 2 Research question 2 of the study sought to investigate and understand the challenges experienced by students during the oral comprehension course. Drawing from our qualitative analysis of the data, the following observations and insights can be reported:

87 Insufficient Textbook Exercises One recurring issue highlighted by the students was the inadequacy of the textbooks used in the course. Students expressed the need for improved textbooks that would provide a greater number of exercises specifically designed to enhance their oral comprehension skills, particularly at the target CEFR level they were expected to achieve. Therefore, it can be inferred that the existing textbooks available may not adequately fulfil the students’ requirements for comprehensive skill development. Request for Increased Instructional Hours Another significant challenge identified by the students was the perceived need for additional instructional hours dedicated to oral comprehension. They believed that allocating more time to this aspect of the course would contribute to their skill acquisition and overall improvement. This finding emphasizes the students’ recognition of the importance of sufficient practice and exposure to oral language to develop their comprehension abilities effectively. Learning through Video Clips Students also expressed a positive attitude towards incorporating video clips as a learning tool to enhance their oral comprehension skills. The students acknowledged the potential advantages of incorporating video content, which combines visual and auditory stimuli, as it can provide a more authentic and immersive language learning experience. Such indicates that the integration of multimedia resources, particularly video clips, has the potential to positively impact students’ comprehension abilities, enhancing their overall language learning process. Exposure to Different English Language Accents Another noteworthy finding was the students’ expressed interest in learning and being exposed to a variety of native and non-native English language accents. The students demonstrated an understanding of the importance of exposing themselves to a wide range of accents in order to effectively adapt to real-life oral communication scenarios.

88 Discussion The findings derived from the survey analysis indicate that the participating students generally held a positive perception regarding their proficiency in comprehending English conversations. This favourable attitude aligns with the outcomes of a previous study conducted by Simons and Colpaert (2015), which also reported a positive reception among stakeholders towards the utilization of the CEFR as a language framework. Furthermore, the survey results shed light on the effectiveness of the CEFR self-assessment scale in providing an accurate representation of students’ language abilities. Notably, a higher level of agreement was observed for self-perceived oral comprehension abilities among individuals classified at lower CEFR proficiency levels. As the CEFR proficiency levels increased, the level of agreement gradually decreased. These findings suggest that the CEFR self-assessment scale can serve as an indicator of learners’ potential achievements. Supporting the present study, prior research conducted by Richard (2020) also affirmed the realistic measurement of students’ language-related abilities using the adapted CEFR self-assessment scale in the Japanese context. An intriguing observation from the study is the misalignment between students’ perception of their English language proficiency in oral comprehension and the intended outcome of the 14-week English language proficiency course, which aimed to produce B2 level learners. Our findings suggest that students did not perceive themselves to have achieved the desired level of competence in oral comprehension, despite the course being designed to facilitate their progression to the B2 level. In congruent to this finding, a previous research also indicated there could be disparities between the learning outcomes and the students’ selfperceived competencies, particularly in the domain of listening sub-skills (Duruk, 2021). Interestingly, the students did not consider accents to be a significant obstacle as long as they were given enough time to familiarize themselves with various English accents. This noteworthy observation can be attributed to the incorporation of listening exercises in the classroom textbooks, which featured recordings of both native and non-native English speakers with diverse accents. By exposing the students to different accents through these exercises, their confidence in comprehending various speech patterns improved. Our research findings are in line with a study conducted by Yen and Thao (2021), which emphasized that extensive listening exercises and exposure can contribute to the enhancement of learners’ listening comprehension skills.

89 Another intriguing discovery from the research is that a considerable number of participants enrolled in the B2 listening class successfully identified themselves as B2 learners. Our discovery corresponds with the results of a prior scholarly investigation conducted by Ahmad Zufrie Abdul Rahman et al. (2021), which demonstrated the effectiveness of the CEFR in nurturing learners who demonstrated a high level of competence in the English language. These findings highlight the positive impact of the CEFR framework on learners’ self-assessment and their ability to gauge their language proficiency accurately. Additionally, the study also identified several challenges faced by students during the CEFR oral comprehension course, including the inadequacy of textbooks in providing exercises for enhancing oral comprehension skills, the request for increased instructional hours dedicated to oral comprehension, the recognition of the value of learning through video clips for an immersive language learning experience, and the students’ interest in exposure to diverse accent variations. Previous literature has explained that the usage of video clips in the classroom could increase learner’s motivation to learn (Laserna & Miguel, 2018). Therefore, our finding highlights the need for improved textbooks, increased instructional time, the integration of multimedia resources like video clips, and opportunities for encountering different accents to address the challenges and enhance students’ oral comprehension abilities effectively. This is seen as important as a previous study illuminated that the CEFR could offer valuable teaching and learning input provided that the framework is utilised to its potential (Kusevska, 2014). In summary, our study unravelled that the students participating in the study held a positive perception of their listening skills in English after having successfully completed the 14-week CEFR oral comprehension course. This positive perception aligns with the results of a bibliometric analysis conducted by Sahib and Stapa (2021), which demonstrated that students perceive the CEFR as having a positive impact on their language learning journey. Conclusion and recommendations The findings of this study shed light on the challenges faced by students during the CEFR oral comprehension course. The inadequacy of textbooks in providing sufficient exercises, the request for increased instructional hours, the recognition of the value of multimedia resources, and the interest in exposure to diverse accents were key insights from the research. Tackling these challenges and incorporating the recommendations discussed below can contribute to enhancing students’ oral comprehension abilities and optimizing their language learning experience.

90 Based on the findings, several recommendations could be made for future research. Firstly, further investigations should focus on developing and evaluating textbooks specifically tailored to enhance oral comprehension skills at different CEFR levels. This would involve designing exercises that effectively target the areas of difficulty identified by students in this study. Secondly, future research could explore the effectiveness of increased instructional hours dedicated to oral comprehension, over the prescribed hours as warranted in The Roadmap (2015). This could involve conducting longitudinal studies to examine the impact of extended practice and exposure on students’ oral comprehension abilities over an extended period. Furthermore, additional research is warranted to explore the integration of multimedia resources, such as video clips, in oral comprehension instruction. Investigating the specific features and design elements that make multimedia resources effective can provide valuable insights into their potential for enhancing students’ comprehension abilities. Lastly, further investigation is needed to explore students’ perception of their language abilities based on other CEFR language modes of reception, production or interaction. Addressing these recommendations in future research endeavors could provide educators and curriculum developers with valuable insights to enhance instructional practices, create more efficacious materials, and improve students’ language related skills within the CEFR framework. This, in turn, would lead to more favourable language learning outcomes and better readiness for real-life communication situations.

91 References Aziz, A., Rasid, R., & Zainudin, W. (2018). The enactment of the Malaysian common European framework of reference (CEFR): National master trainer’s reflection. Indonesian Journal of Applied Linguistics. doi:10.17509/ijal.v8i1.9989. Beacco, J.-C., Byram, M., Cavalli, M., Coste, D., Cuenat, M. E., Goullier, F., ... & Vollmer, H. J. (2016). Guide for the development and implementation of curricula for plurilingual and intercultural education. Council of Europe. Beaven, A., & Sadler, R. (2016). Beyond the European Language Portfolio: Enriching the self-assessment grid for language learner autonomy. In G. Dettori & I. Candelier (Eds.), Autonomy in Language Learning and Teaching: New Research Agendas (pp. 17-40). Council of Europe Publishing. Borhany, M., Tahriri, A., & Tous, M. D. (2015). The impact of explicit/integrated instruction of listening comprehension strategies on EFL learners’ L2 listening comprehension and their overall strategy use. Journal of Applied Linguistics and Language Research, 2(8), 128-144. Chong, Gloria, and Hamidah Yamat. Teachers’ Implementation of CEFR-aligned Curriculum: a Preliminary Study. Journal of English Language Teaching and Applied Linguistics, vol. 3, no. 3, 2021, pp. 5-9, doi:10.32996/jeltal.v3i3.1366. Chen, H., Hong, Y., Liu, Y., & Peng, J. (2022). Technology-enhanced listening instruction in higher education: A systematic review. Computers & Education, 185, 104989. Chen, X., Chen, C., & Zhao, Y. (2020). Active listening in college English classroom: A phenomenological study. Higher Education Studies, 10(2), 25-36. Chin, C. K., & Brown, C. M. (2019). Active listening strategy: Facilitating students’ comprehension of lecture input. TESOL Journal, 10(2), e00332.