Assessing the Assessment: using psychometrics to establish reliability and validity of portfolio assessment

Medicine (Project course) 2011-09-02

Abstract

In summary, previous studies of portfolios in medical education have tested inter-rater reliability by the use of varying methods without problematizing the difference between inter-rater agreement (IRA) and inter-rater reliability (IRR), thereby neglecting the choice of the best method. Furthermore, reliability has predominantly been tested with two assessors, a setting not fit for medical education with a large number of students and assessors. Although validity by some authors is regarded as a dynamic process and specific for a certain process (2), different aspects of validity of the portfolio deserves a better discussion. In a recently published paper on a portfolio pilot, undergraduate students’ reflections on professional issues in general practice were qualitatively analysed (37). The authors were able to show sufficient results of reliability and internal validity of the portfolio by the use of criteria for qualitative research, and content validity was also supported by the content of students’ reflections. However, the inter-rater reliability, the concurrent validity, and possible effects of the self-selected sample of students warranted further psychometric evaluation. This study aims to analyse these criteria, using different traditional psychometric methods.