Our Goal

Apply the principles and techniques of science and engineering to study and improve learning, especially of expertise.

Major Accomplishments

Bold words refer to categories in the top menu bar.

Italicized words are names of subsections or pages; go directly there using the search box.

Prior to starting RELATE, the PI and his son Alex wrote the platform mycybertutor.com and contents that became MasteringPhysics.com – see MasteringPhysics. As described in Genesis of RELATE, the PI then decided to start an education group to study and improve it.  RELATE’s goal of improving education using science and engineering, involves a great many different approaches, each with different facets.  The most important ones form the basis of the acronym RELATE which is the group’s name:

REsearch in Learning, Assessing and Tutoring Effectively

In this brief introduction, we describe these areas of concentration and show their interconnection.  This introduction serves as an overview of the entire corpus of RELATE’s activities and serves as an introduction to this web site.

Online educational technology generates a time-stamped log of each student’s clicks and submissions, perhaps a megabyte of data per student per semester.  This is about 10,000 times the amount of data in the gradebook of a traditional course, and it opens the possibility to doing research into student actions and learning and to improving individual instructional resources.  RELATE seeks to use these data to understand and improve the learning of students. This necessitates Research to extract, instrumentalize, and display features that might influence learning.  The first challenge is that obviously interesting features like “time on task” for each resource require thoughtful definition and programming to extract meaningful numbers.

Called data mining, this process also includes features like student strategies, affect, and habits.  We found several surprises: that most students have over 30% of their weekly homework done two nights before it is due; that students vastly prefer video help when doing homework, but revert to the textbook on (open book) exams; and that copying homework is by far the best predictor of final exam grades.

RELATE’s research has also included surveys of teachers and students and experiments to answer fundamental questions like, What should we teach and How much will they remember at graduation?  Too often these important questions are neglected and teachers simply cover the syllabus in their college catalog.

Applying the principles of science and engineering requires quantitative conclusions and comparisons based on reliable measurements.  In education, measurement is called Assessment.  Assessment drives instruction by providing students, teachers, and content authors with feedback on whether they’re meeting their goals. ‘Summative assessment’ like examinations focusses students’ attention and effort, and strongly influences what learning the teacher tries to engender.  RELATE’s objective in assessment is to gain deeper insight into the student’s thinking, behavior, both to increase basic understanding and also for real time ‘formative assessment’ for improving instruction. Towards these ends we have made several advances and applications of Item Response Theory (IRT), including making a measure of current ability and how it depends on the student’s path through the resources in MasteringPhysics.com.  In order to gain more detailed insight into a student’s current knowledge, we are pioneering multi-dimensional IRT analysis to gain insights beyond current (overall) ability and final grade.  In addition, we have made Special Assessments that assess specific areas of expertise.  These include Mechanics Reasoning Inventory, single topic quizzes, and angular procedures tests for procedures like finding torque, rotational energy, and moment of inertia.

Given these detailed assessments, RELATE can study Learning – defined as the improvement from one assessment to another.  After introducing a couple of Models of Learning, we study Learning in entire courses; trying to determine which type of resources (etext, online homework, lecture preparation, recitation attendance,..) give the most learning per unit of student time, whether less skillful students learn as much as more skillful students, whether MOOC students learn more or less than on-campus students using the same materials in a teacher-led course, and how does course structure (e.g. frequency of quizzes) change student behavior.

We also study Learning within courses, usually with random controlled studies (i.e. with a control and an experimental group).  We showed that doing symbolic or numerical problems helped improve performance on a similar conceptual question administered afterwards, but not vice versa. We also studied the effects of adding a diagram to a physics problem ( To draw or not to draw?_), the amount by which tutor-like responses to specific wrong answers reduced the time and fraction of wrong answers on a subsequent problem, and we showed that kinesthetic learning caused quicker learning of simple procedures than the same problems delivered in multiple choice format.

RELATE was formed to study data from the Socratic homework tutor written by the PI and his son Alex Pritchard, originally named mycybertutor.com (this has become Pearson’s MasteringPhysics.com, M’chemistry.com, M’Engineering.com, etc.).  Offering comments on particular wrong answers, a selection of hints upon request, and follow-up comments and questions after completion of problems, it emulates a Socratic tutor, the epitome of Tutoring.

Subsequently RELATE moved to other online learning environments with the objective of putting more of the instructional resources into one location to broaden our understanding of student learning.  Using the platforms LON_CAPA, MITx, and edX, we analyzed both on-campus courses and MOOCs (Massive Open Online Courses).  We developed an “Integrated Learning Environment” (now it would be called an online ‘blended’ course) containing both instructional and interactive classroom activities, with assessment across all activities.

RELATE’s goal statement ends with the pedagogical goal, ‘especially of expertise”.  Our interest is more specifically in helping students become more expert problem solvers in mechanics.  Even more specifically we seek to impart ‘strategic knowledge’ – the ability to analyze a problem to determine which already learned concepts and laws can help reach the solution.  We have developed of MAPS Pedagogy (Modeling Applied to Problem Solving) that Improves problem solving expertise in Newtonian mechanics.  It has demonstrated three major successes:

• improves the expertise of student’s learning attitudes as measured by the Colorado Learning Attitudes towards Science survey by about 10%,
• raises student scores on the MIT final exam by a standard deviation, and
• improves their scores in the subsequent Electricity and Magnetism course by ~ ½ a standard deviation.

RELATE ran several instances of its Mechanics online MOOC using and developing this pedagogy.

RELATE didn’t intend to study Cheating, however upon discovering that online homework copying is by far the dominant (anti-)predicter of learning.  Online Cheating Prevents learning: students who copied over 50% of their problems had a 55% chance of failing to pass Physics 1 and Physics 2 on schedule.  Separately, we found that cheating was rampant in (our) MOOCs: even though the certificates were of no academic value, we found that 7% of certificate earners “harvested” more than 20% of their answers from fake accounts they set up with false names.  So prevalent is copying that we showed that copiers distort studies of online learning (e.g. their high homework scores but low exam scores distort the correlation between doing homework and learning).  Our surveys showed that copying written homework is even more prevalent than copying online homework, and that students nationally self-reported nearly twice the copying of our MIT sample, and had significantly less moral condemnation of most types of cheating behavior.  This emphasizes that academic dishonesty was a serious educational problem even before the recent arrival of cheating companies like Chegg.com and CourseHero.com from which students can find answers to homework and exam questions.

Support

The RELATE project is supported by the NSF, NIH, and Google. It has also been supported by MIT as follows:

• the James Ferry Fund
• the Class of `55 funds
• the MIT physics department
• the MIT UROP program, and
• the Dean for Undergraduate Curriculum.

We are grateful to Alex Pritchard and Adam Morton for adopting many of our suggestions for their myCyberTutor product.

Drafting compass in RELATE logo based on original artwork by abluescarab (A. Gilston), available under Creative Commons Attribution 3.0 License

Needless to say, we are seeking additional support to enable this ambitious program to continue. If interested in supporting us, please write to Prof. David Pritchard.