Volume IV: What works, what matters, what lasts

Calibrated peer review

21st Century Pedagogies

Download the PDF version of this document.

Calibrated Peer ReviewTM (CPR) was developed on a research-based model. What do scientists and engineers do? Research begins with proposals. Scientists and engineers write research proposals and review peer proposals. Scientists and engineers do research and write and peer review research manuscripts. Peer review plays a prominent role in the progress of science. Writing with anonymous peer review is thus the model for Calibrated Peer Review. CPR is a comprehensive, web-delivered instructional tool, which not only promotes student understanding through writing but also develops student critical thinking skills through the process of evaluation and reviewing. Because the program is web-delivered, computer-managed, and uses peer review, instructors can use the CPR program with any class size; students can do their assignments where ever they have web access; and administrators do not need to provide additional teaching resources for grading.
http://cpr.molsci.ucla.edu

How CPR Works

Calibrated Peer Review comprises an integrated set of network tools that manages the submission and evaluation of written student work. As in traditional classes, in classes using CPR, students begin an assignment by composing a document based on an instructor’s guidelines. Rather than turning in a printed paper, however, the written work is submitted electronically. After the students submit their work, the CPR program guides them through a tutorial on peer reviewing of that particular assignment. Students then receive detailed feedback on their evaluation skills . Only when students become competent reviewers by passing the calibration exercises in the tutorial, do they evaluate their peers’ work in a double-blind anonymous process to ensure privacy.1 None of the work, however, is anonymous to the instructor. Throughout an assignment the instructor has access to all student work and can monitor and assess class and individual student progress. In the final step in a CPR assignment students return to their own essays and, using the same criteria they have used for their peers, evaluate their own work. After the end of an assignment, CPR prepares and presents the student with an individual report, which shows all of his or her reviewers' assessments and comments. In the background, CPR prepares an instructor’s report, which yields detailed information about class and individual performance. The instructor can view any document and any review; he or she can re-evaluate any student’s work.

CPR provides two other key features: an assignment authoring tool and an assignment library. The authoring tool enables the instructor to create the components of the writing and reviewing activity that form the assignment. The assignment library, created and published to the web site by other faculty in the discipline, lets an instructor by-pass the hours of development time it takes to craft a well-written assignment and select a relevant topic for his or her class.

The instructor has two choices in using CPR: He or she can author new assignments or use existing library assignments. Only imagination limits the CPR assignments that faculty create. Any topics instructors want their students to delve into deeply can be developed into an effective assignment. The CPR Assignment Authoring Tool facilitates the task and ensures that all the components of a complete assignment results.

Rather than prepare their own assignments, instructors can opt to use one in the CPR library. They simply select the assignment, set the class parameters for timing and grading the assignment and inform the students. The next step is to examine the student performance and results.

What CPR Enables

CPR enables frequent student writing in any discipline at any level without overloading the instructor. It has worked effectively in classes as small as twenty students and in classes as large as 500 students. In addition, students learn to read for content—they get reinforcement. By the time students have read seven documents on the same theme, they have the content down pat. One can, in fact, observe that students answer the content questions much more quickly toward the end of an assignment than they did at the beginning. The quality time on task that students spend while using CPR is apparent to even first time faculty using the program (Hollis 2002). Students also develop key skills such as abstracting, persuading (proposals), developing logical arguments, describing, assessing, criticizing, analyzing, and reviewing. These are key higher-order thinking skills (Bloom 1956) that routine exercises never develop. In CPR assignments, students encounter engaging ideas and ponder important issues. Most important, through writing and reviewing, students develop critical thinking (Bean 1996; Falchikov and Goldfinch. 2000).

non-STEM fields Assignments
composition & writing 193
economics & management 58
medicine 41
communications 13
high school 13
education 9
history 9
business law 6
nursing 3
others 66
 
STEM fields Assignments
chemistry & biochem 430
life sciences 199
science writing 67
geology 42
math & statistics 30
ethics 23
psychology 19
physics & astronomy 17
science education 14
engineering 12
computer science 7
Table 1: Distribution of CPR assignments by discipline (March 2004)

CPR Impact on Student Learning

Studies at three different universities in chemistry, biology, and economics independently document that students taught using CPR assignments perform approximately 10% better on traditional course exams than students taught through traditional lecture and text-book methods (Chapman, 2000; Russell, 2001; Pelaez, 2002). This finding holds for all levels of student ability and does not depend on whether the exam questions are essay-based or multiple-choice. Reviewing also causes learning gains. Duplicated studies show that the reviwing process significantly enhances content understanding for the weakest students beyond their learning gains from the essay writing (Murphy, http://www.uctltc.org/news/2001/12/feature_6.htm). A national study of 10 courses in multiple disciplines shows that students’ ability to review and evaluate content for accuracy, argument, and logic improves regardless of the whether CPR assignments are implemented at the high school, community college, comprehensive college, or research university (Russell, 2003).

At the end of the Molecular Science Project in 2001, the user-base consisted of 101 colleges and universities. In these institutions, CPR had served more than 520 courses enrolling more than 16,000 students. The library consisted of approximately 175 completed assignments. The use has grown exponentially since then.

The CPR user-base now consists of 500 institutions. CPR has been used in more than 1900 courses, which have enrolled over 72,000 students. In addition, the library now consists of 1275 completed assignments representing an enormous national scholarship of teaching from faculty who are using CPR in their courses. Table 1 summarizes this creative effort (Russell, 2004).

Calibrated Peer Review is currently delivered from a UCLA server and is offered to the academic community without cost. Any instructor may “register” through the home page web site. Technical assistance is provided through e-mail and a listserve. Workshops are given frequently to inform faculty on ways to use the program, and on authoring techniques to develop well-crafted assignments that enhance student learning.

References:

Bean, J.C. Engaging Ideas, The Professor’s Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom Jossey-Bass, San Francisco (1996)

Bloom, B.S. (Ed) Taxonomy of Educational Objectives, Cognitive Domain Longmans, Green and Company, New York (1956)

Chapman, K. Impact of CPR in Economics 200, California State University, Northridge April 2000 (private communication)

Falchikov, N., Goldfinch, J. “Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks” Review of Educational Research, 70, 3, 287-322 (2000)

Hollis, K. (UC Riverside, Chemistry) ‘What is clear to me from the questions that I have taken is that the students are engaged with the topic! It is presented in a form that they like (i.e. they get to play with their computers) and it is a very directed study forum. I wish that I had hundreds of problems presented through an Internet interface like CPR, covering the entirety of the material in my course. Computers and the Internet have been around long enough and there is enough familiarity now, that the technology no longer seems to be a hindrance to study. Instead, it engages the students through a medium with which they are familiar and through which they are receptive to instruction and learning.’ E-mail correspondence (2002)

Murphy, P., “Collaborative Innovations: New UC-Wide Partnerships Are Transforming the Teaching and Learning Experiences” http://www.uctltc.org/news/2001/12/feature_6.htm.

Pelaez, N., “Problem-Based Writing with Peer Review Improves Academic Performance in Physiology,” Advances in Physiology Education, 26, pp174-184, (2002)

Russell, A. A. The Evaluation of CPR, prepared for HP e-Education; Business Development, UCLA (2001)

Russell, A. A. Writing About Chemistry Using Calibrated Peer Review,TM 225th ACS National Meeting, New Orleans, LA (March 2003)

Russell, A. A. Calibrated Peer Review – A Writing and Critical Thinking Instructional Tool, Invention and Impact, NSF-CCLI Conference, Crystal City, VA, (April 2004)


1 Lightfoot, J.T. A Different Method of Teaching Peer Review Systems, Advances in Physiology Education, 274, 6, S57-S61 (1998).