|
|
|
|
|
Journal of the Experimental Analysis of Behavor, 2007, 88, pp.103-113
Rasil Warnakulasooriya, David J. Palazzo & David E. PritchardWe studied students performing a complex learning task, that of solving multipart physics problems
with interactive tutoring on the web. We extracted the rate of completion and fraction completed as
a function of time on task by retrospectively analyzing the log of student–tutor interactions. There was
a spontaneous division of students into three groups, the central (and largest) group (about 65% of the
students) being those who solved the problem in real time after multiple interactions with the tutorial
program (primarily receiving feedback to submitted wrong answers and requesting hints). This group
displayed a sigmoidal fraction-completed curve as a function of logarithmic time. The sigmoidal shape is
qualitatively flatter for problems that do not include hints and wrong-answer responses. We argue that
the group of students who respond quickly (about 10% of the students) is obtaining the answer from
some outside source. The third group (about 25% of the students) represents those who interrupt their
solution, presumably to work offline or to obtain outside help.
Physical Review Special Topics-Physics Education Research, 4, 010102 (2008)
Young-Jin Lee, David J. Palazzo, Rasil Warnakulasooriya and David E. Pritchard.We investigate short-term learning from hints and feedback in a Web-based physics tutoring system. Both the skill of students and the difficulty and discrimination of items were determined by applying item response theory IRT to the first answers of students who are working on for-credit homework items in an introductory Newtonian physics course. We show that after tutoring a shifted logistic item response function with lower discrimination fits the students’ second responses to an item previously answered incorrectly. Student skill decreased by 1.0 standard deviation when students used no tutoring between their incorrect first and second attempts, which we attribute to “item-wrong bias.” On average, using hints or feedback increased students’ skill by 0.8 standard deviation. A skill increase of 1.9 standard deviation was observed when hints were requested after viewing, but prior to attempting to answer, a particular item. The skill changes measured in this way will enable the use of IRT to assess students based on their second attempt in a tutoring environment.
Submitted to Physical Review Special Topics-Physics Education Research.
David E. Pritchard, Young-Jin Lee & Lei BaoWe present quantitative learning models predictions of student’s knowledge vs. amount of instruction – that are based on assumptions motivated by various theories of learning: Tabula Rasa, Constructivist, and Tutoring. These models predict the improvement (on the posttest) as a function of the pretest score due to intervening instruction. We introduce a connectedness model with a connectedness parameter which interpolates between Tabula Rasa and Constructivistbased assumptions. Over a wide range of pretest scores on standard tests of introductory physics concepts, it fits high quality data nearly within error. We suggest that data from MIT have low connectedness (indicating memorybased learning) because the test used the same context and representation as the instruction, and that more connected data from the University of Minnesota results from instruction in a different representation from the test.
Submitted to Am. J. Phys.
Elsa-Sofia Morote and David E. PritchardIn an MIT calculusbased introductory Newtonian mechanics course we study the effectiveness of various instructional course elements: electronic and written homework, collaborative group problems, and class participation. We measure effectiveness by the slope of the regression line between a student’s score (used as a proxy for participation) on a particular course element and his normalized gain on the various assessment instruments. These instruments were the MIT final exam comprised mainly of multipart problems demanding analytic responses, and two widely used standard physics tests that emphasize conceptual knowledge: the Force Concept Inventory and the Mechanics Baseline Test. The results show that interactive course elements are associated with higher gains on assessment instruments: doing interactive electronic homework administered by myCyberTutor correlated with large gains on the final exam producing a learning effect of 1.8 ± 0.4 standard deviations on the final examination score. MyCyberTutor and collaborative group problem solving correlated with gains on the more conceptual tests. We also report surveys that demonstrate that students have had an increasingly favorable opinion of myCyberTutor over the four terms of its use.
We study the usefulness of hints in learning physics using myCyberTutor (aka: Mastering Physics), a web-based homework tutor. About 48% of the students request hints on average to a given problem with an average of one hint request per wrong answer submitted. Administering related problem pairs to two equally skilled groups in different orders we find that the group which solves a problem in a given problem-pair second requests on average 12% fewer hints than the group which solves it first. The maximum reduction (19%) in requests for hints occurs for a problem after solving its related tutorial problem. These results support the cognitive theory that feedback is a form of information that helps students in learning.
We study two pedagogies using myCyberTutor, a web-based homework tutor. Two groups, tutorial-first (TF) and problem-first (PF), taking introductory mechanics solve a tutorial and a related problem pair. The TF group solves the tutorial problem before solving the related problem(s) while the PF group solves the related problem first. The TF group has significantly less difficulty on the related problem(s) and completes them more quickly than the PF group. The PF group shows a smaller reduction in the difficulty they have with the self assessment questions in the tutorial. The reduction in difficulty of the TF group on the related problem is twice as much per unit of time spent on the tutorial as is the corresponding reduction in difficulty experienced by the PF group. We conclude that schema acquisition is better facilitated by the tutorial-first approach.
We study the effectiveness of various instructional course elements (electronic and written homework, collaborative group problems, and class participation) in an MIT calculus-based introductory Newtonian mechanics course. We measure effectiveness by the correlation coefficient between a student’s score on a particular course element and the gain of that student’s grade on assessment instruments. These instruments were the MIT final exam and two widely used standard physics tests that emphasize conceptual knowledge: the Force Concept Inventory and the Mechanics Baseline Tests. The results show that interactive course elements have significantly higher correlation with increased assessment scores: interactive electronic homework administered by myCyberTutor correlate significantly with gain on the final exam, and that myCyberTutor and collaborative group problem solving correlate most strongly with gain on the more conceptual tests. We also report surveys that demonstrate that students have had an increasingly favorable opinion of myCyberTutor over the four terms of its use.
Progress Report 2004: Research Laboratory of Electronics, Massachusetts Institute of Technology.
Progress Report 2002: Research Laboratory of Electronics, Massachusetts Institute of Technology, ISSN 0163-9218.
Students in a calculus-based mechanics course at MIT were given the conceptual Force Concept Inventory (FCI) test both at the start and the end of the course. Gains in student scores are correlated with individual use of the different components in the course: small-group tutorial sessions, written problem sets, an interactive web-based problem set, collaborative work and number of sessions viewing multi-media content on the web. This makes it possible to study the relative effectiveness of the different components in improving conceptual understanding. The data indicate that usage of the webbased problem set led to gains in FCI score that were twice as large compared to the written problem set. Solving complex problems in collaborative groups also led to gains in FCI score whereas both the small-group tutorial sessions viewing web-based multi-media were less effective.Internal report - Department of Physics, Massachusetts Institute of Technology, 2000.
We demonstrate learning and problem-solving transfer within the web-based homework tutor MasteringPhysics by considering time to completion, the number of hints requested, and the number of incorrect responses given. The group of students who were prepared by a prior related problem solves a related follow-up problem in ~14% less time on average compared to an unprepared group on that problem. Furthermore, the prepared group requests ~15% fewer hints and makes about ~11% fewer errors on average than the unprepared group.(Proceedings of the 2005 Physics Education Research Conference, P. Heron, L. McCullough, and J. Marx, (Eds.) pp. 41-43, ISBN 0-7354-0311-2 )
We demonstrate learning and problem-solving transfer between related physics problems using the Socratic web-based tutor, MasteringPhysics. Two equally skilled groups of students taking introductory mechanics solve related physics problem pairs in reverse order with respect to each other. We show that for problems containing help (“skill builders” and “self tutoring problems”) in the form of requestable hints, descriptive text, and feedback, twice as many students were able to complete problems correctly in real time compared to problems that did not provide any help (end-of-chapter problems). The group that did a problem second (prepared group) in a given related pair was able to solve it in ~15% less time on average compared to the group that did the same problem first (unprepared group). Furthermore, the prepared group requests 7% fewer hints on average than the unprepared group. We conclude that learning and problem-solving transfer is facilitated through skill builders and self tutoring problems, which also are pedagogically superior to end-of-chapter problems.Conference Proceedings: EdMedia -World Conference on Educational Multimedia, Hypermedia & Telecommunications Vol. 2005, 2005, pp. 2976-2983.
We discuss new measures in assessing the skill level of students based on their interactions with the Socratic homework tutor, MasteringPhysics. We show that the measurement of the difficulty of problems for a given student as determined by the time to first correct response, the number of incorrect responses without advice, and hints have high reliability (96%). As an immediate application, we demonstrate how item difficulty can be used to construct an item discrimination measure that would result in predicting the final exam score with a correlation of 0.634.Conference Proceedings: EdMedia -World Conference on Educational Multimedia, Hypermedia & Telecommunications Vol. 2005, 2005, pp. 2523-2529.
Time to completion reveals problem-solving transfer
Rasil Warnakulasooriya & David E. PritchardTwo equally skilled groups of students taking introductory mechanics solve related physics problem pairs in reverse order with respect to each other, using the web-based Socratic tutor, MasteringPhysics. In tutorial problems containing help in the form of requestable hints, descriptive text, and feedback, twice as many students were able to complete problems correctly in real-time compared to problems that did not provide any help (end-of-chapter problems). The prepared group in a given related pair was able to solve it in ~15% less time on average compared to the unprepared group. Furthermore, the prepared group requests ~7% fewer hints on average than the unprepared group. We conclude that shorter completion times and problem-solving transfer are facilitated through tutorial problems.(Proceedings of the 2004 Physics Education Research Conference, J. Marx, P. Heron, and S. Franklin, (Eds.) pp. 205-208, ISBN 0-7354-0200-0)
Technology Closes the Gap between Students' Individual Skills and Background Differences
Elsa-Sofia Morote & David E. PritchardWe examine the correlation of 12 background variables determined from a student survey with assessment instruments including paper-testing instruments (final exam and weekly tests) and an electronic assessment from the results of a Web-based homework tutorial. Several of the initial background variables showed correlation with hand-graded weekly or final paper tests. Level of previous math and physics courses taken correlated with positive results on both. However, none of the background variables correlated with the student’s performance on the more reliable Web-based tutorial homework score. On this basis we suggest that the use of un-timed Web-based tutorials can make an important contribution to assess student performance, without bias due to students’ background differences.Conference Proceedings: Society for Information Technology and Teacher Education: International Conference, Vol. 2004, Issue. 1, 2004, pp. 826-831.
Reliable Assessment with CyberTutor, a Web-based Homework Tutor
David E. Pritchard & Elsa-Sofia MoroteSocratic electronic homework tutor, CyberTutor can integrate effectively instruction and assessment. The following study shows that CyberTutor assessment has about 62 times less variance due to random test error than a three-hour final examination, and 53 times less variance than twelve weekly tests. This huge improvement results from the combination of three factors. First, the typical student interacts with CyberTutor for at least 48 hours per term, 16 times longer than the 3-hour final exam and 10 times more than 12 weekly tests together, diminishing the effect of lucky guesses and careless errors. Secondly, by considering requests for hints, solutions, and importantly the number of wrong answers and the time the student takes to complete each part of the problem, CyberTutor can make a much better determination of the student’s skill on a problem. Lastly if a problem is too difficult for the student as presented, CyberTutor’s hints and spontaneous responses to wrong answers systematically adapt the difficulty of the problem to the student’s skill level, and the CyberTutor assessment algorithm accurately determines that level.World Conference on E-learning in Corporate, Government, Health, & Higher Education, Vol. 2002, Issue. 1, 2002, pp. 785-791.
CyberTutor, A Socratic Electronic Homework Tutor at MIT
David E. Pritchard, Elsa-Sofia Morote & David KokorowskiCyberTutor is an interactive web-based program that functions as a personal tutor for the student, offering help with the problems when requested or in response to incorrect answers. CyberTutor collects extensive feedback which is processed to inform the teacher about current performance and lack of foundational skills of each student, to identify troublesome problems and concepts for the class, and to inform the problem author about flaws in the problems. CyberTutor contains problems involving Newtonian Mechanics at the freshman college level and at high school AP level, many of which involve conceptual issues as well as quantitative ones.World Conference on E-learning in Corporate, Government, Health, & Higher Education, Vol. 2002, Issue. 1, 2002, pp. 2711-2712.
Inductive Influence of Related Quantitative and Conceptual Problems
Phillip Dukes, David E. Pritchard & Elsa-Sofia MorotePhysics education research shows that conceptual understanding is not necessary for students to do well on standard quantitative problems in introductory physics. This study addresses a related question: is conceptual understanding sufficient or helpful for students to do well in quantitative problems? We approached this question by splitting a class of approximately 100 students into two equal skillful groups, A and B. We gave the A group a conceptual problem prior to giving a related quantitative problem. If the A group experiences less difficulty with the quantitative problem than the B group, we say that the conceptual problem has positive “inductive influence” on the quantitative problem. By administering the same conceptual problem to the B group after they do the quantitative one, our methodology also addresses the reverse question “How much conceptual knowledge does a student learn from working through a quantitative problem?” Our results show little inductive influence upon the quantitative one from doing the conceptual one first, but suggest that working the quantitative problem first helps students with the subsequent conceptual one.Educational Resources Information Center (ERIC), ED463978
National Association for Research in Science Teaching – NARST- 2002 Conference, New Orleans.
What Course Elements Correlate with Improvement on Tests in Introductory Newtonian Mechanics?
Elsa-Sofia Morote & David E. PritchardEducational Resources Information Center (ERIC), ED463979
National Association for Research in Science Teaching – NARST- 2002 Conference, New Orleans.
Inductive Influence of Related Quantitative and Conceptual Problems
Phillip Dukes & David E. PritchardWe have preliminary data on the extent to which answering conceptual questions on a topic helps with a subsequent quantitative problem on the same topic and vice versa. Our research was performed using CyberTutor, which administers free response questions with multiple layers of hints and subparts, and judges proficiency based on the number of hints and wrong answers submitted on the way to the solution. We divided a class of 100 students into two equally skilled groups and administered the conceptual problem and then the quantitative problem to one group, reversing the order of presentation to the other group. Alternatively we administered variants of a problem with the conceptual parts before vs after the quantitative part. Our preliminary data shows very little positive correlation in either direction. In contrast, we find strong inductive support of related problems that demand numerical responses vs analytical responses: the number of errors on the second problem can be reduced by up to 40% (four standard deviations).Proceedings of the 2001 Physics Education Research Conference, S. Franklin, J. Marx, and K. Cummings (Eds.) pp. 53-56, ISBN 0-9716536-0-7