Authors
Kiran Gawali, Education Insight and Evaluation Analyst, FoNS ETL
Helen Walkey, Learning Analytics Product manager, Imperial College Learning Analytics Team
The FoNS EdTech Lab was asked to help the Department of Life Sciences evaluate the success of/value in switching to a new project marking workflow. This case study is an example of an evaluation of a process change and describes how they planned what to investigate and how.
The switch involved using an assessment tool (WISEflow) that was new to students, markers and the programme administrator.
The first step was to identify the objectives of changing the marking workflow and how these objectives would be met using the tool and different processes. Step two was to define what success would look like if each objective was met and then to work out how that success could be measured or assessed. These steps were entered into an evaluation table, as shown below.
Table with examples of objectives, actions to meet the objectives, success indicators and metrics.
|
Department/Module/Activity: Life Sciences PG Project Marking LIFE97012 |
|||
|
Step 1a. Objective |
Step 1b. Action How will the objective be met? |
2a. Success Indicators What would it look like if the objective was met? May need to break down objectives & action |
2b. Metrics How could that success be measured? |
|
To reduce the admin burden on the programme administrator. |
Use of a tool with a simpler workflow and fewer manual steps for the programme administrator. |
All administrative processes are managed in a single platform. No redacting of comments required by the programme administrator (required for previous workflow). Programme administrator satisfied with the management of feedback comments in the system.
|
Count of flows in WISEflow compared to Turnitin assignment drop boxes in the previous workflow. Feedback from the programme administrator in a semi-structured interview. |
|
For it to be easy for students to submit project work and get feedback. |
Provide guidance for students on how to submit via WISEflow and access feedback |
Students found it easy to submit work and access feedback on the platform. Low number of requests for technical help from students. |
Student feedback survey. Count of student queries received in FoNS EdTech Lab mailbox and categorisation of assistance requested. |
Lessons learnt:
Understanding the context for the change and talking with staff in the department about what success would look like were important when planning the evaluation.
It was a rounded evaluation because it looked at the user experience as well as the technical aspects.