Categories

Archive

Disclaimer

De meningen ge-uit door medewerkers en studenten van de TU Delft en de commentaren die zijn gegeven reflecteren niet perse de mening(en) van de TU Delft. De TU Delft is dan ook niet verantwoordelijk voor de inhoud van hetgeen op de TU Delft weblogs zichtbaar is. Wel vindt de TU Delft het belangrijk - en ook waarde toevoegend - dat medewerkers en studenten op deze, door de TU Delft gefaciliteerde, omgeving hun mening kunnen geven.

Meta

Digital exams – essays and short answer questions

Posted on 31/05/2013 by

Now that TU Delft invested into creating large computer rooms for digital exams, it is of the essence to make optimal use of these rooms. Still the majority of the exams are paper based and not all exams can or should be converted into automatically marked exams. Seeing the positive response of students towards computer exams I wrote about in my previous blog post, I decided to search for tools for taking exams existing of essay and short answer questions, combined with functionality on easy onscreen grading.

I got inspired by Roel Verkooijen’s presentation about Checkmate at a meeting of the Special Interest Group Digital Assessment. Working with this system, the exam itself is still taken on (special) paper, the review of the exams is done onscreen. The paper exams are scanned. In the reviewing screen, the question, the model answer and the scanned student response are shown and the reviewer can add comments. The system keeps track of the progress of reviewing and facilitates working with different reviewers. Students can review the grading online and even discuss the comments made within the same system. I find that the major disadvantage of Checkmate is that this system still relies on student’s handwriting.

Checkmate is tailormade for Erasmus MC, so unfortunately not available to other institutions.

At the moment I’m experimenting with CODAS. This is text grading software that ranks documents according to their level of similarity with certain selected  documents; to what degree do they resemble good examples or the poor examples. With CODAS an instructor only needs to grade about 30 to 50 assignments in order to get a reliable ranking of the documents. CODAS can also perform a plagiarism check. The experiment I’m doing is about finding out what kind of exam questions, or  better, what kind of short answers can be revised reliably with this system. CODAS requires digital input (*.txt format), put the system does not provide that itself.

My search for assessment systems that specialize in open and short answer questions led me to a a JISC-study on Short answer marking engines. This study discusses the use marking engines for short questions. Two of these engines I would like to discuss:

  • C-rater by Educational Testing Service (ETS) uses natural language processing (nlp) to assess a student’s answer. It can be used to grade open questions that require a converging answer. It can not handle questions were  a student’s opinion or examples from the student’s own experience is required. Even though C-rater corrects for misspelling, use of synonyms, paraphrasing and context it does not reach the same level of reliability of human marking. The exam designer and the answer model builder need to be quite knowledgeable on the C-rater’s mechanisms in order to get a good result. Another drawback is that the system needs up to 100 scored responses to build the answer model. The system is still being improved.
  • ExamOnline by Intelligent Assessment Technologies (IAT) uses marking schemes (model answers) that are added by human markers to assess a student’s response. The system provides an exam delivery system and facilitates onscreen grading. The grading is a combination of computer and human marking. Another interesting feature this software offers is the possibility to add hand-drawn sketches to a digital exam. When a sketch answer is required, the system shows the student a unique code that the student needs to fill in on the paper sketch. Afterwards, when the sketches are scanned the system automatically connects the scan to the student’s exam so the grader can assess the sketch onscreen.

I am very curious what OnlineExam can offer, I hope to be able to experiment with this software to see if it might meet our needs.

On top of that, in 2 weeks I hope to get a sneak preview of MapleTA’s new features on essay questions. The current version of MapleTA (8.0) provides a reasonable editor to students answering an essay question, but the grading screen is by far not practical for onscreen grading of short answer questions exceeding 30 words.

Be Sociable, Share!
Posted in Meta Keijzer-de Ruijter   permalink 

1 comment

© 2011 TU Delft