Automated formative and summative feedback: Mechanical Engineering problem sets

Assessment overview

These assessments focus on automated feedback provided for mathematical, short-answer questions to problem sets for undergraduates. The assessment feedback is both formative and summative.  

The self-study exercises consist of homework sheets set frequently throughout the autumn and spring term with accompanied feedback The summative assessment consists of ‘progress tests’ worth 5% of the 5 ECTS module (these are “low stakes” assessments), set at the end of the first term. There are 17 formatively assessed homework sheets in total across the module, which are set weekly in autumn and spring termThe formative self study exercise questions include a range of types, from basic practice, to reinforcement, to extended challenges. They are not designed to ‘test’ understanding, but rather to give students the necessary exercise to learn. 

Design Decisions

Rationale for automated marking

Student homework sheets were not marked in the past on this module, so automation allows feedback on formative assessments where previously there was none. More specifically, it automates ‘low level’ feedback that stops students getting stuck on questions, and allows them to go deeper into their homework before requiring help. The subsequent contact time with teachers is higher quality due to deeper discussions. See this article for more information.

Automation for summative assessments improves staff time management at a busy time of year and increases the speed and consistency of feedback to students.  

The reason for the low stakes assessment is two-fold; it is an entry point to the technology for the teacher (allowing development of the system), and it introduces students to this type of assessment and feedback without a high mark penalty if they get things wrong. The progress tests are a good vehicle for this innovation as they are summative (so students take them seriously), but the weighting is low. 

Rationale for the design of the Lambda software

Initially the assessments were hosted on Mobius platform. However, the following issues were identified 

  • General user experience (student and staff) 
  • Restriction on programming language to program feedback (Maple only). 
  • Restriction on error-carried forward capability. 
  • Restriction on feedback capabilities - because it is ‘grading’ focussed. 
  • Restriction on analytics. 

The Lambda feedback software was designed to meet the needs that Mobius couldn’t. In addition to that the software will soon allow the teacher to parameterise a question, so that the exact content seen by a student is unique to them and so the correct answer is different for each student. There are obvious issues with the integrity of remotely-delivered summative tests, so this system allows teachers to change some (or none) of the questions, which prevents students colluding on their answers.  

Questions design

For the self-study exercises, the same considerations were made as for any worksheet-style assessment, including, from the student perspective: the purpose of the assessment e.g. which Intended Learning Outcome is being assessed (what concept, what skill, what knowledge, what misconception?); how long will it take to complete?; How clear are the instructions for the assessment? How much interest/engagement does it generate for the student? How easy is it to navigate through the assessment, e.g. how good is the user experience?  

From the staff perspective, the teacher needs to consider how to construct a question to enable generation of the formative automated feedback, so must consider the structure of the question, and any potential ambiguity in the answer, e.g. use of different symbols for mathematical concepts.  

Fit with other assessments and the programme/ module

The formative feedback assessment is new and innovative; the feedback provided does not cause  any known disruption to any other module, and supports students’ learning of essential concepts in a way that is beneficial to the higher level modules that they take in later years in their degrees.   

For the summative assessments, changes (such as from paper based to computer based and from manually marked to automatically marked) were applied across the whole cohort to ensure consistency of the marking.  

Practicalities

Preparing students for assessment

As preparation for the formative assessments students are given general advice on study methods and feedback literacy. Feedback literacy is a two-way thing between students and teachers: students need to receive a lot of feedback before they get skilled at knowing how to react to it; teachers need to provide quality feedback. Students can check their understanding of the module content and concepts during bi-weekly tutorials with staff.  

As preparation for the summative assessments students are given a briefing in a lecture and are given written information about it on Blackboard; they are directed towards the ILOs, the syllabus and the module descriptor. They also take a readiness test (to check they can use the software). For one cohort (who were particularly stressed) a mock test was run too. Students are encouraged to complete the formative worksheets as preparation for the summative assessment.  

Monitoring student progress

As all the assessments are online, useful analytics can be easily generated to track student engagement, e.g. showing peaks in online activity on the day of the lecture; showing that many students access the homework sheet as soon as it is set, then engagement tails off; that there is a gradual decline in student engagement through the time, but that overall students do keep working and keep trying to catch up.  

Feedback arrangements

Feedback for both formative and summative assessments is automated, as discussed above in the explanation of the Lambda feedback software. It is completely consistent and objective, but also manually checked. Any changes that need to be made (e.g. if an error is spotted in a question) are applied algorithmically to the whole cohort. Students can ask questions about their feedback during tutorials throughout term, or in a dedicated tutorial feedback session in January, after the summative assessment has been completed.  

For the summative assessment the feedback is not instant, as teachers check the grading before releasing marks. Feedback is currently limited to grades (marks) with no comments. 

Online adaptations

The summative tests were designed to be delivered online, and can be taken remotely (they are currently taken in-person, in computer rooms, but worked well entirely remotely during the pandemic). The online mode of assessment hasn’t affected the design of the assessment, except that it increases the importance of parameterised questions (which are bespoke for each student) for the summative test. 

Hear what the Imperial Experts have to say...

What is summative and formative assessment?

What is summative and formative assessment?

What is summative and formative assessment?

Kate Ippolito, Centre for Higher Education Research and Scholarship

Disadvantages of formative assessments

Disadvantages of formative assessments

Kate Ippolito, Centre for Higher Education Research and Scholarship

How to avoid over-assessing students

How to avoid over-assessing students

Judith Webster, Registry

The value of attaching credit to formative assessments

The value of attaching credit to formative assessments

Kate Ippolito, Centre for Higher Education Research and Scholarship

What skills to exams develop?

What skills to exams develop?

Katie Dallison, Careers Services

How to help students frame employability skills

How to help students frame employability skills

Katie Dallison, Careers Services

Overview

Faculty: Engineering
Department: Mechanical Engineering
Module name: MECH50010 Fluid Mechanics 2
Degree: Mechanical Engineering MEng
Level: Year 1 and 2
Format: Homework sheets and progress tests, online
Approximate number of students: 200
Delivery mode: online; both remote and on-campus
Weighting and credit: Summative assessment is 5% of module (low stakes)
Module ECTS: 5
Module type: Core

More information

Interviewee: Peter B Johnson 
Role: Principal Teaching Fellow