Investigating DIF: The role of response format for students following the syllabus for Swedish as a second language

Experiences from the Swedish National Tests in Social Studies

Abstract

In developing the national tests in social studies, a critical issue is ensuring that the items work in the same way for subgroups of students, i.e., the test is fair and shows no signs of severe Differential Item Functioning (DIF). In Sweden, the national tests will be digitalized, but the proportion of items that can be assessed automatically, which requires a selected response format (SR), is also intended to increase. This study aims to analyze how different items (defined by varying formats and contents) function for students following the syllabus of Swedish as a second language (SvA) compared to the general student population. The study is based on data from four years of investigation from the Swedish national tests in social studies (2018, 2019, 2022, and 2023). Analyses based on Item Response Theory (IRT), focusing on DIF, are conducted. The results reveal that there are differences between the two syllabus groups: Students following the SvA syllabus have more difficulties with SR items, covering factual and conceptual knowledge aspects. However, SvA students perform better on constructed response items where they are asked to describe, analyze, and elaborate on a topic using their own language. The results are indeed important, not least in the context of the upcoming digitalization of the national tests. The SvA student group generally achieves lower results on the national tests in social studies, on average. As this study shows that the SvA group has significantly more difficulties compared to the general student group with the SR items that are intended to increase due to digitalization, there seems to be an obvious risk for this group to be disadvantaged. From a subject didactical perspective, this may provide opportunities to strengthen the factual and conceptual knowledge of SvA students. From a test constructor perspective, the results stress the importance of considering both content and format in DIF analyses.