Updating the algorithms powering Pair and Order Rank
The most common question we get from users is to explain the logic behind the Glicko-based scoring system on OpinionX. While Glicko (the Bayesian rating system popular in online multiplayer gaming) is a pretty robust and scalable system for our Pair Rank and Order Rank questions, researchers were often undermined by the inability to explain how opinions were scored and ranked by our 'black box' ranking engine.
Ultimately, we felt that this downside outweighed the benefits of Glicko as a scoring system, so we've rebuilt Pair and Order Rank blocks on two new ranking formulas. Our focus has been to make it much easier to understand and explain how OpinionX calculated the scores for your opinions so that you can ensure that your research projects have a higher chance of succeeding in generating impact.
The two new methods — Dowdall Borda Counts for Order Rank and Win Rate Analysis for Pair Rank — are both explained in greater detail (with screenshots and examples) here: New Ranking Formulas (April 2023 Product Update)
Note: this update does not change how participants experience your existing OpinionX surveys, only how the ranked scores are calculated on the backend. The update went live this morning (Apr 12th).