Peer review ‐ how to set up

This page describes how peer reviews can be set up for exercises. Peer review is a useful alternative grading method to automatic or teacher grading.

What is a peer review

In the peer review process, students grade other students answers to the same question they have answered themselves. The grading is based on peer review questions defined by the teacher. Peer review is a useful alternative grading method to automatic or teacher grading.

How to set up a peer review exercise

1. Add a new exercise.

See Add new exercise wiki page for detailed instructions

2. Configure peer review

• Add peer review: toggle peer review on / off.
• Add self review: If enabled: Once student has submitted the required number of peer reviews, they will review their own exercise submission using the peer-review questions. Self-review does not affect the points the student receives from the exercise.
• Use course default peer review config:
  • On: the exercise uses the shared courses default peer review settings. This is useful if all peer reviews follow the same format, for example students always answer the same set of questions.
  • Off: the peer review settings only apply to this particular question. This is useful if the peer review questions only apply to this one question.
• Peer reviews to receive: This number determines how many peer reviews the student needs to receive for their answer before they can receive points for the exercise task.
• Peer reviews to give: This number determines how many submissions by other students the student needs to grade before they can receive points from the exercise task. (This number needs to be higher than the peer reviews to receive to make sure that every students receives enough peer reviews, taking into account that some students may drop out of the course and not submit peer reviews).
• Peer review accepting strategy:
  • A Automatically accept or reject by average:
    • PASS: If the average score of the likert scale questions that the student receives is above the peer review accepting threshold: student receives full points for the exercise.
    • FAIL: If the average score is below the threshold: student automatically receives 0 points.
  • B Automatically accept or manual review by average:
    • PASS: If the average score of the likert scale questions that the student receives is above the peer review accepting threshold: student receives full points for the exercise.
    • FAIL: The answer will be moved to the answer requiring attention section, where the course staff grades the answer. Student does NOT receive points automatically.
  • C Manual review everything: All answers are moved to the answer requiring attention section after they have received enough peer reviews.
• Points are all or nothing: See section of this page called Points: All or nothing and weighted questions
• Peer review accepting threshold: Students either pass or fail the exercise based on the average of grades they receive from the Likert scale type peer review questions. If the average is above the threshold, the student completes the exercise successfully and get points (if accepting strategy is A or B). If the average is below the threshold: student does not receive points (accepting strategy A) or the answer will be moved to the answer requiring attention section, where the course staff grades the answer (accepting strategy B). NOTE: This setting does not apply to accepting strategy C.

Example: If student A receives these scores:

• Peer review question 1 (likert scale): 3 (from student C), 5 (from student Y): Average 4
• Peer review question 2 (likert scale): 2 (from student C), 4 (from student y): Average 3
• TOTAL AVERAGE: 3.5
• RESULT: 3.5 > 2.1 (Peer review accepting threshold set for the exercise task), which means that Student A's answer is passes the check and receives points.

3. Add peer review questions

• Add peer review question button: Adds a new peer review question
• Peer review question type dropdown: Select question type (likert scale, essay). Likert scale questions are required if you want to use automatic scoring for the exercise task.
• Peer review question: This text is shown to students when they peer review other students answers. Should include the criteria for grading the answer.
• Answer required: Determines whether an answer must be given or not.

Points: All or nothing and weighted questions

• Points are all or nothing is enabled: On the platform, the default scoring for peer review exercises is that the students either receive full points for the exercise or no points.
• Points are all or nothing is disabled: If you disable the all or nothing scoring option, you can set weights to peer review questions. The number of points students can receive in this case varies anywhere between the full points and no points at all (instead of just the options of no points or max points).

Set up weighted questions

1. Uncheck the Points are all or nothing option in the exercise settings

2. Set the weight for each likert scale type peer review question. The weight is a value between 0 and 1. The total weights for likert scale type peer review questions should add up to 1. For example:

• Q1: weight: 0,3
• Q2: weight: 0,2
• Q3: weight: 0,5 *TOTAL = 1.0 (0,3 + 0,2 + 0,5)

3. Save your changes.

Formula for determining points when Points are all or nothing is disabled

Student B evaluates student A's submission.

The likert values that Student A receives from student B are multiplied by the weight of the question. Peer review Q1: 5 (likert-value, 1-5) x 0.25 (weight of the question) = 1.25 Peer review Q2: 4 x 0.25 = 1 Peer review Q3: 5 x 0.25 = 1.25 Peer review Q4: 5 x 0.25 = 1.25

Values are added together values added together 1.25 + 1 + 1.25 + 1.25 = 4.75

Calculate average of total values received from all students doing the peer review: 4.75 (Student B ) + 3.25 (student C) + 3.25 (Student X) = 11.25 11.25 (total value) / 3 (number of students who have peer reviewed the essay) = 3.75

Convert the total value received from students into percentage 3.75 (average from all students) / 5 (number of possible likert values) = 0.75 (or 75%) (percentage on likert scale)

Multiple percentage with the maximum number of points available for the exercise: 0.75 (percentage on likert scale) x 4 (max points) = 3 points

Student A would receive 3 out of 4 points for the submission to the exercise.

Peer review flow - from a student point of view

1. Student A submits an answer to an exercise. The answer is usually a text of a certain length.
2. Student A starts the peer review process, they are randomly assigned another students submission to peer review. For example, students B and X.
3. Student A reviews student B's answer - usually by answering likert-scale questions.
4. Student A reviews student X's answer.
5. Student A waits until they have enough peer reviews from other students.
6. Student A receives points for the exercise based on the Peer review accepting strategy.

Tip: Add example answers for students to peer review

We recommend that you and other course staff submit sample answers to peer review exercises. The number of sample answer should be the number of minimum peer reviews to give. This way students who are first to complete the exercises have answers to peer review immediately after they have submitted their own answers and do not have to wait for answers by other students. The sample answers do not need to be perfect or ideal answers, but instead something similar to what students might submit.

Special cases

• Not enough peer reviews received: If the student does not receive enough peer reviews for their answer: The answer will be automatically moved answers requiring attention category (in other words, to be graded by the teacher) after 21 days (3 weeks).
• Teacher does not grade in time: If the teacher does not grade an answer in the answers requiring attention category within 90 days (3 months), student will automatically receive full points for the answer.

Questions and answers:

• Question: When does a submission start to receive peer reviews?

  • Answer: As soon as the student has submitted their answer.

• Question: When are the points for a submission calculated?

  • Answer: The points are calculated after the student has submitted the number of peer reviews required by the exercise.

• Question: Do all received peer reviews affect the points the student receives for their exercise submission?

  • Answer: No, only the peer reviews received until the required number of peer reviews to receive is met. For example, if peer reviews to receive is set to 3, the points the student receives for the exercise are based on the first three received peer reviews. Additional peer reviews do not matter.

• Q: Is it possible to set a minimum peer review length?

  • A: No. Currently it is not possible to set a minimum length for peer reviews. You can instruct the students to write a peer review of a certain length but this cannot be programmatically checked by the platform.