Exploring the use of text-only and multimode feedback in scientific assessment tasks: quantitative analysis

For my MEd project, I conducted a case study to examine the experiences of students and their marker with two different feedback delivery methods (text-only and multimode), using both quantitative and qualitative analytical methods. Here I will focus on the quantitative analysis. The design was quasi-experimental, with the same students receiving feedback from the same marker via the two different delivery methods. This allowed comparisons to be made between the two methods. I also explored the experiences of two subgroups of students – those with English as an additional language and those whose grade was lower than their expectations.   

Quantitative analysis of student questionnaire data was undertaken in R using reproducible workflows for each of the analyses to avoid introducing errors through manual processing and to allow the code to be re-run as required; this approach had the added benefit of facilitating checks on the data and the assumptions made in each analysis. Because of the ordinal nature of the data collected, non-parametric statistical tests were used: the Wilcoxon signed-ranks test for paired samples, e.g. when comparing students’ responses to the questions on text-only feedback versus the questions on multimode feedback, and the Mann-Whitney test for unpaired samples, e.g. when comparing responses of students whose mark was below expectations with those whose mark was not below expectations. I followed the widely used convention of describing results where p £ 0.05 as “statistically significant” but, being mindful of the controversy around this practice, I reported the exact p-value in each case, together with the test statistic and effect size. Effect size is inflated when sample size is small and when a quasi-experimental approach is used. As both these factors applied in my study, I was hesitant to include effect-size values, because they could be misleading if compared with values from other studies. In the end, I decided to include the values for completeness and to allow comparisons to be made within the case study, but not to describe the effects in words (‘small’, ’medium’, ‘large’, etc) based on criteria such as Cohen’s. Although I used statistical analysis to explore my findings, the goal of case study research is to attempt to understand findings in a general sense rather than to seek statistical certainty.