Analysis and Quality Assurance
McKenna and Bull (2000) have provided some important quality assurance recommendations:
- Integrate the scheduling of computer-based tests into the timetabling for end-of-module examinations.
- Ensure the proper moderation of CAA examinations, as for traditional examinations.
- Consider appointing an additional external examiner with expertise in the construction and presentation of CAA.
- Incorporate feedback mechanisms which guide academic staff in the improvement of tests and systems.
- Ensure that staff have been offered and have attended the relevant staff development sessions.
- Develop a procedure which defines and checks that question banks have been supplemented with a percentage of new questions each year.
- Verify that piloting procedures and question analysis (to ensure reliability and validity) have been undertaken.
- Establish an upper limit on the amount of CAA examination per module. (For example, in order to encourage lecturers to offer a balanced assessment profile to students, the use of CAA might be capped at 40 per cent of the total module mark).
- Agree standards (in terms of screen design, instructions within test, function of buttons) to guarantee consistency in presentation of tests thereby minimising student confusion.
- Integrate a programme of evaluation covering all aspects of the system.
Following e-assessment it is important to determine whether the questions have been effective at discriminating between students and determining whether or not the questions have assessed what they were professed to be assessing. There are two measures that should be calculated for objective test questions, and these are often provided automatically by e-assessment software, the facility value and discrimination. The facility value is the fraction of students making a correct response to an item. Discrimination is how well an item discriminates between able and less able students, as measured by performance on the whole test.
Analysis of these statistics is important because alongside the examination results they can help academics determine which questions should be retained, and which should be altered or removed (Zakrzewski and Steven, 2003).
Feedback on the conduct of the test should be sought from the students and peers as well as from the external examiner, and the student performances should be evaluated to provide feedback on the teaching that has been occurring in the course. Johnstone (2003) suggests that a useful indicator is to look at changes in facility values. If the values are lower, then it probably relates to student ability, if the values are the same except for a particular topic then it's more likely to be related to the teaching.
Zakrzewski and Steven (2003) and McKenna and Bull (2000) suggest that effective and robust quality assurance procedures are required to successfully implement e-assessment for summative purposes.
