Embedding AI into assessment

Assessment overview

This case study involves a project based assessment where groups of MBA students are working together in teams to build the first part of a business plan for a start-up with the domain of Artificial Intelligence. During the 5 week module the students are scaffolded through a series of shorter written exercises that build towards their capstone submission. The use of AI is embedded into one of the written exercises where students are asked to critique an AI generated output of their business strategy. 

Design decisions

Overview of the assessments on the module

The aim of the module is for students to explore applications of AI in finance, health and other markets and to build new business models, products or technical concepts. The final product of the module is a pitch of a new AI venture that they wish to launch in either a corporate setting or as an independent start-up.  

 The module is purely based on coursework which comprises of the following: 

  • The final report of 10-20 slides of a written submission and the final presentation on the group output (that resembles a Dragon’s Den style verbal pitch 
  • Written individual assignment elaborated where students are asked to use AI to critically evaluate its output about the originality of their business idea.  
  • Peer evaluation and group project contribution that are graded 
  • Quizzes to ensure that students are doing the required work and engaging in class.  
  • Class participation  

Rationale for the inclusion of an AI component

One of the coursework assessment, the individual written assignment, was redesigned in response to the rise of large language model systems like ChatGPT. Before the popularisation of Chat GPT the task read: 

Following your problem identified in Written Assignment 1, please write a short essay of 200-500 words on the topic of “what is unique about your solution”. As before, you may potentially (but are not required to) describe the same unique attributes that differentiate your solution from others that might exist, but you should provide your own individual perspective on the answers. 

This task was changed once ChatGPT became widely used to include a deliberate use of an AI tool (students weren’t exclusively restricted to using ChatGPT but ended up choosing it as their preferred tool): 

Following your problem identified in Written Assignment 1, you will again be collaborating with your LLM-AI. You will be submitting 2 items in this written assignment: 

  1. Instruct your LLM-AI of choice to generate a 250-word explanation of what is unique about your AI solution. Your written assignment is to critique this output (250 words +/-). What was good about the result? What did not work well? What would you do differently? 
  1. Submit the venture description output that the LLM-AI itself generated. Please identify which LLM-AI you used. 

Rationale for the group component

Group component is the fundamental learning design for this type of module. Most of the learning that students will gain from particularly an entrepreneurship and business plan focused class is peer learning. Therefore figuring out how to operate as part of a team where you have to take a complex task and subdivide it into smaller tasks, navigate domains of status and control, authority, expertise, ability to work with others is what's necessary to succeed in running a start-up.   

Fit with other modules

This is an elective module that interacts seamlessly with another elective module - Entrepreneurial Journey. Students who take the AI ventures module can pick up their work on their business plans in Entrepreneurial Journey and extend it. They'll get a more rigorous and in depth look at the topic and also more rigorous and in depth set of feedback at end of term.  

Practicalities

Preparing students for AI use

Prior to undertaking the assessment the students were shown how use a LLM. This was done through composing AI poetry in class, i.e. the students used AI to compose short poems about AI ventures that were then read in class. The reason for it was making sure that every student had an account set up to enter prompts and generate an output. Through this, students learnt how to use the tool in a non-threatening format. 

Preparing students for group work

Students at the start are asked to sign an honour code of agreement where they agree to contribute to group work. Outside of that there is no preparatory work done to ensure students are well prepared to work in a group. This is because of the nature of the students who tend to have some working experience before going into the MBA and the nature of the programme that is group work heavy. 

Assigning groups

Groups are assigned according to interests. A Google sheet is created with students ideas for businesses and others interested in a similar idea can sign up and form a team together. This helps to ensure students are driven by their interests and are passionate about the business, which assigning students to groups would not help to achieve. 

Peer marking and feedback

Group contribution are peer marked. This way the group self regulates without the need to implement top-down interventions. 

There is no formal preparation for peer feedback and peer marking, again due to the nature of the programme and the students.  

Marking arrangements

Each component described in assessment overview has weighting attached to it and is formally marked. The distribution is as follows: 

Final report and the pitch 

50% 

Written individual assignment 

10% 

Peer evaluation and group project contribution 

20% 

Quizzes  

10% 

Class participation 

10% 

 

Provision of feedback

Feedback on the individual written assignment was provided in the form of a sentence or two. The students tended to fall into patterns of response so it was easier to generate feedback that can apply to more than one student. The individual answers were different, but they bucketed nicely into handful of categories - did the student understand that ChatGPT was repetitive and superficial? Did they just spot that it was superficial and not repetitive? Did they just spot that it was repetitive and not superficial? Did they fail to extend the analysis of the output into something they would use in their final submission? Even though there was only one marker involved in marking a substantial amount of submissions, this approach to feedback allowed to lower the feedback burden yet for students to still get useful information they can apply in a timely manner.

Online adaptations

This assessment could easily be implemented both in an Online class setting or in person. 

Hear what the Imperial experts have to say...

Leadership and followership

Leadership and followership

Leadership and followership

Katie Dallison, Careers Services

Different ways of assessing group work

Different ways of assessing group work

Dr Iro Ntonia, Centre for Higher Education Research and Scholarship

Successfully using exemplars

Successfully using exemplars

Dr Iro Ntonia, Centre for Higher Education Research and Scholarship

Pros and cons of using exemplars

Pros and cons of using exemplars

Dr Iro Ntonia, Centre for Higher Education Research and Scholarship

What are exemplars?

What are exemplars?

Dr Iro Ntonia, Centre for Higher Education Research and Scholarship

What is authenticity?

What is authenticity?

Professor Martyn Kingsbury, Centre for Higher Education Research and Scholarship

How to design authentic assessments

How to design authentic assessments

Professor Martyn Kingsbury, Centre for Higher Education Research and Scholarship

Pros and cons of group work

Pros and cons of group work

Dr Iro Ntonia, Centre for Higher Education Research and Scholarship

Why do employers value group work?

Why do employers value group work?

Katie Dallison, Careers Services