Each italki teacher has a "star" rating from 0 to 5 stars based on student feedback.

If the student does not leave a rating for the lesson, the rating for the lesson will be 5 stars by default.

italki uses a Bayesian Average to determine a teacher's rating. This is a different calculation than the "normal" average or mean. It is a weighted average that is often used on sites where individual people receive ratings from others.

The reason we use the Bayesian Average

The problem we have found is that teachers who just joined italki are disproportionately affected by the ratings from their early students. The feedback from a teacher's first few students might not really represent them well.

For example, if a new teacher's first student gives her 5 stars, does that mean she is really a 5-star teacher? Or, if that student did not like the lesson and gives 1 star, then is that what should show on her profile?  This is what would happen if star ratings were calculated according to the basic average.

Every teacher will inevitably run into a student that is not happy with their lesson. This happens, and it does not necessarily mean that the teacher or student is bad. When a teacher has taught 100 lessons, where 99 lessons got a 5 star rating, and 1 lesson got a 1 star rating, the basic average is fine. The basic average (4.9+) shows that the teacher consistently gives great lessons.

However, what happens if the teacher’s first student was the one unhappy student? The teacher's basic average rating would start at 1.0, and then slowly rise up to the long-term average of 4.9+. Unfortunately, we've found that that teacher would have a hard time getting a second lesson. Many students would not be willing to take that chance, even though we know the teacher is ok.

Bayesian Average

What the Bayesian Average does is reduce the impact of outliers when there is limited data.

When there are only a few data points, the Bayesian Average discounts the basic average, relative to the average of all the teachers on italki. As the teacher receives more ratings, there is more confidence that the teacher's basic average is accurate. Over time, as the teacher earns her own ratings, the Bayesian Average converges with the teacher's basic average.

You can think of it roughly this way:

italki accepts teachers that we believe will teach good lessons for italki students. When the teacher has just a few lessons on italki, the teacher's rating comes partly from her own student feedback and partly from the average rating of all teachers on italki together. As the teacher "earns" more of their own feedback, then the teacher's rating relies more and more on her own feedback and less on the rating of other teachers. 

Here are some examples of how the ratings may change over time:

Example 1: Consistently “Perfect” Teacher

As the teacher continues to get 5 stars for every lesson, so her rating starts out affected by the site average, but gradually increases to show the consistent 5-star ratings.

Example 2: Consistently “Bad” Teacher

The influence of of the site average balances out the first few lessons, but because the teacher continues to get a "2" rating, the rating falls.

Example 3: "A Bad Start" Teacher

This is what happens when a new italki teacher has a bad start or an accidental bad review. Without the Bayesian average, the teacher's rating would look much worse. But because the next lessons got good ratings, the teacher's rating will climb back up towards 5.

Example 4: "Improving" Teacher

Similarly to example 3, if a teacher starts out good but improves their ratings even more, then the ratings for the first 10 lessons might look something like this.

References:

You can learn more about Bayesian Averages on Wikipedia:

http://en.wikipedia.org/wiki/Bayesian_average

This article also contains a great explanation of why Bayesian averages are a good choice for online ratings and how to calculate them: Of Bayesian Average and Star Ratings

http://fulmicoton.com/posts/bayesian_rating/

Improvements Are Coming

No rating system is perfect, and the italki team is thinking of better ways to show teacher quality. If you have suggestions, contact us and let us know!