Assessment

We have developed an assessment that has ~ 100 times less testing error than traditional final examination and has the added advantage of being integrated within the tutoring process. It is based on assessing the process of the student working through an interactive problem and not just the correctness of his/her final response. The reduced testing error allows the reliable determination of the student's skill on a large number of sub-topics within the curriculum without the necessity of distracting tests. These assessments are based on variables that are available only within an online tutorial environment (or direct student-teacher interaction), including the number of hints, wrong answers, and solutions requested, and the time to completion – none of this information is available within traditional testing environments. The powerful insight from this detailed assessment allows the prediction of the scores on standard tests with a predictive validity coefficient of 0.77. This enables students to monitor their progress on the scale of the test they are studying for, and it allows teachers to raise the scores of their classes on standard high stakes tests.

Figure 3 Testing error is determined by dividing the test (or the tutoring problems) into two equivalent subtests (e.g. even and odd numbered problems). A highly reliable instrument would measure the same score on each equivalent subtest, and the points would lie on the diagonal. The reliability of the final exam is 85% (i.e. 15% of the observed variance is due to testing error) whereas the electronic homework had a reliability of 99.76% (i.e. only 0.24% of the observed variance is due to testing error) - Data from MIT course 8.01 in Spring 2001. (see Reliable assessment with CyberTutor, a web-based homework tutor - Pritchard, D., Morote, E. S.).