How are teacher ratings calculated?

italki uses a Bayesian Average to determine a teacher's rating. This is a different calculation than the "normal" average or mean. It is a weighted average that is often used on sites where individual people receive ratings from others.

A Bayesian Average reduces the impact of outliers when there is limited data.

When there are only a few data points, the Bayesian Average discounts the basic average, relative to the average of all the teachers on italki. As the teacher receives more ratings, there is more confidence that the teacher's basic average is accurate. Over time, as the teacher earns her own ratings, the Bayesian Average converges with the teacher's basic average.

Each italki teacher has a "star" rating from 0 to 5 stars based on student feedback.

If the student does not leave a rating for the lesson, the rating for the lesson will be 5 stars by default. The rating for trial lessons will not be counted for the average rating on the teacher profile.

The reason we use the Bayesian Average

The problem we have found is that teachers who just joined italki are disproportionately affected by the ratings from their early students. The feedback from a teacher's first few students might not really represent them well.

If a new teacher's first student gives her 5 stars, does that mean she is really a 5 star teacher? Or, if that student did not like the lesson and gives 1 star, then is that what should show on her profile?  This is what would happen if star ratings were calculated according to the basic average.

Every teacher will have a student that is not happy with their lesson. It does not necessarily mean that the teacher or student is bad. When a teacher has taught 100 lessons, and 99 of those lessons had a 5 star rating, and 1 lesson had a 1 star rating, the basic average (4.9+) shows that the teacher consistently gives great lessons.

However, what happens if the teacher’s first student was the one unhappy student? The teacher's basic average rating would start at 1.0, and then slowly rise up to the long-term average of 4.9+. Unfortunately, we've found that teachers have a hard time getting a second lesson in this situation. Many students are not willing to take a chance, even though we know the teacher is ok.

Bayesian averages and italki Teachers

italki accepts teachers that we believe will teach good lessons. When the teacher has just a few lessons on italki, the teacher's rating comes partly from her own student feedback and partly from the average rating of all teachers on italki together. As the teacher "earns" more of their own feedback, then the teacher's rating relies more and more on her own feedback and less on the rating of other teachers. 

Here are some examples of how the ratings may change over time:

Example 1: Consistently “Perfect”

As the teacher continues to get 5 stars for every lesson, so her rating starts out affected by the site average, but gradually increases to show the consistent 5-star ratings.

Example 2: Consistently “Bad”

The influence of the site average balances out the first few lessons, but because the teacher continues to get a "2" rating, the rating falls.

Example 3: "A Bad Start"

This is what happens when a new italki teacher has a bad start or an accidental bad review. Without the Bayesian average, the teacher's rating would look much worse. But because the next lessons got good ratings, the teacher's rating will climb back up towards 5.

Example 4: "Improving"

Similarly to example 3, if a teacher starts out good but improves their ratings even more, then the ratings for the first 10 lessons might look something like this.

References

You can learn more about Bayesian Averages here. 

This article also contains a great explanation of why Bayesian averages are a good choice for online ratings and how to calculate them.