Washback Effect From Changes to Comprehensive Assessment Program Math Test for Junior High School Students Author: Ming-Chiu Chang (Research Center for Psychological and Educational Testing, National Taiwan Normal University), Li-Ying Huang (Research Center for Psychological and Educational Testing, National Taiwan Normal University), Chia-Jung Chen (Research Center for Psychological and Educational Testing, National Taiwan Normal University), Po-Hsi Chen (Department of Educational Psychology and Counseling, Research Center for Psychological and Educational Testing and Institute for Research Excellence in Learning Sciences,  National Taiwan Normal University), Fen-Lan Tseng (Research Center for Psychological and Educational Testing and Institute for Research Excellence in Learning Sciences,  National Taiwan Normal University)

Vol.&No.：Vol. 67, No. 1
Date：March 2022
Pages：227-254
DOI：https://doi.org/10.6209/JORIES.202203_67(1).0008

Abstract：
Purpose
This study investigated the effects of incorporating constructed-response items into the math tests in Taiwan’s Comprehensive Assessment Program for Junior High School Students (CAP). Using questionnaires, this study explored the differences in junior high school math teachers’ overall approaches to teaching and compared the assessment approaches used in preparation for the Basic Competence Test for Junior High School Students (BCT) with those used for the CAP.
Literature Review
2.1 Washback Effect
Hughes (2003) defined the washback effect as the effect of a test on teaching and learning, which may be positive or negative. Studies on the washback effect have been classroom centered, focusing on the curriculum, materials, pedagogy, assessments, feelings, and attitudes.
2.2 Possible Effect of Adjusting the CAP
The purposes of the CAP and BCT are different; therefore, the functions of the exams, scoring methods, types of test items, test times, and presentation of the test results differ. For a comparison of the CAP and BCT, please refer to the CAP website (https://cap.rcpet.edu.tw/index.html).
The goal of adding constructed-response items to the CAP math test was to induce positive measurement-directed instruction, with the expectation that teachers would adjust their teaching and assessment methods according to the type of items on the CAP. The Ministry of Education, county and municipal education bureaus, and junior high schools attach great value to the CAP results; thus, teachers must respond to the expectations of school authorities and use various strategies to help students prepare for the examination, which may put pressure on teachers.
Therefore, although the CAP has been administered for many years, whether math teachers can select appropriate and diverse forms of assessment in their regular teaching in response to the incorporation of constructed-response items is worth exploring.
Methods
3.1 Participants and Procedure
A survey was conducted with an online questionnaire, which was sent to all junior high schools, and three math teachers from each school were selected to complete the survey. After completing a general math teacher questionnaire, teachers with more than 5 years’ teaching experience were asked to complete a math backwash questionnaire to ensure that those completing the questionnaire had teaching experience related to both the BCT and CAP.
3.2 Measures
This study used two types of teacher questionnaire and semistructured in-depth interviews as follows:
(1) Questionnaire on the washback effect of the math test: This questionnaire had eight items and 35 subitems mainly investigating the curriculum, teaching materials, pedagogy, and assessment used by math teachers in preparation for the BCT and CAP.
(2) General math teacher questionnaire: This questionnaire surveyed teachers’ opinions regarding the CAP and their use of the Feedback “Report on Monitoring the Academic Quality of the CAP” in each school.
(3) Semistructured in-depth interviews: Nine junior high school math teachers were asked in semistructured interviews for their comments on the results of the first two questionnaires.
Results
(1) Approximately 70% of the teachers stated that they would spend more time on the instruction of test content covered by the BCT and CAP.
(2) The teachers used textbooks as the main source of instruction for both exams, with additional practice of constructed-response items and sample questions from previous exams for the CAP (approximately 7.5% higher).
(3) Most teachers used one-way lectures or two-way teacher-student techniques to prepare students for both exams.
(4) The teachers placed the greatest emphasis on reading comprehension skills for the CAP, followed by communication skills.
(5) No significant difference was detected in the proportions of item formats on regular midterm and final exams, even with the slight increase in constructed-response items.
(6) Over 50% of the teachers thought the change from a norm-referenced to a criterion-referenced scoring system would not reduce students’ test anxiety, and nearly 70% of the teachers believed that the new scoring system would reduce learning motivation.
The findings of our study indicate a washback effect from the incorporation of constructed-response items into the CAP math test in junior high school math classes.
Conclusions and Suggestions
5.1 Conclusions
The characteristics of the math teachers’ preparation for the CAP and BCT are compared as follows:
(1) As the CAP items became more authentic and competency-based, teachers placed more emphasis on students’ reading comprehension skills.
(2) The addition of constructed-response items to the CAP led to adjustments in the types of test items and scoring methods used by teachers, with a greater emphasis on constructed-response items; this emphasis on constructed-response items was reflected in the teaching content.
Although the BCT contains no constructed-response items, most teachers nevertheless prepare students for this type of item; regular assessments now include constructed-response items. Such changes in the content of instruction achieve the intended purposes of adding constructed-response items to the test, that is, to guide teachers’ instruction, strengthen students’ ability to communicate and express themselves in math, and make students’ learning of math more comprehensive.
5.2 Suggestions
(1) Teachers should use the data from CAP test reports effectively to identify the test items on which students performed the most poorly and to further understand students’ learning blind spots.
(2) In addition to teaching mathematical knowledge, teachers should strengthen students’ reading comprehension and communication skills to enhance their mathematical literacy.
(3) To enhance the learning motivation of basic level students, a certain percentage of basic or easy items can be used in class to build confidence. Learning activities can be arranged in accordance with student abilities so that students of different ability levels can experience success in math.

Keywords：washback effect, high-stakes test, the Comprehensive Assessment Program for Junior High School Students (CAP), mathematic constructed-response items

《Full Text》