Thanks for your input. Let’s work through your objections.
We first have to agree that skill of an individual player can be estimated by the outcome of the games he played in. Otherwise, we would have to reject any automated ranking, which would be impractical.
Given that, we assume that each end score has an unknown probability distribution of which the skill of the player is but a single variable. Therefore our estimate must take into account the outcome of many games if we aim to isolate the effect of the player. In that respect you are right. However, only a relatively small number of games are necessary as the average distribution of each score will quickly approach a Normal distribution (the famous bell curve). In the absence of bias for team formation, the average opponent team’s score would follow a Normal distribution centered around the expected team score over all possible team formations (something Anet can estimate with high accuracy as they have access to all games) while the player’s team score would follow a Normal distribution centered around an expected score highly correlated to the player’s skill. A 100 games may seem a small number but I assure you that it is sufficient from a statistical standpoint IF there is no bias in team attribution AND the population remains constant.
To address your concern, we could wait until a player has participated in a certain number of games in a division, say 25, before he can be eligible to move up. That way, his estimated skill would be impacted by the actual conditions of the division and we could compare it with some confidence against the skill of that population. After 100 games without moving up, the player’s estimated skill could be said to be an accurate reflection of his rank within that division.
About the luck of matchmaking. There isn’t any. The current algorithm is clearly not a random assignment within the population of the division. If that were the case, there would not be such long winning or losing streaks. Therefore, we have to assume that there is always a possibility of bias introduced by matchmaking and we have to neutralize it otherwise the outcome of matches are not independently distributed and the above argument does not work anymore (no bell curve, no way to estimate skill with any degree of confidence). That’s why it is so important to discriminate between wins and losses and give them equal weight in any estimate of the player’s skill.
You say that the average losing score is almost always around 300-350 range. I say that this is to be expected over all games and in no way an argument against using score as an estimate. To the contrary, the average score of losing games involving a particular player would be tightly distributed around a precise value that is highly correlated to that player’s skill. That is the precise property upon which I base my suggestion.
Regarding the Foefire map, you raise a valid point. Killing the opposite lord can cause a large score difference. However, I argue that players take that into consideration when they pick a strategy. You will occasionaly see a team reverse the outcome of a match by giving up defending the caputure points and go for the opposite team’s lord. In any case, any difference in score distribution by game types will average out and approach a Normal distribution after a number of independent games. Therefore I cannot accept your conclusion. There could be a bias if players focused on specific games types, but then you could just take that into account, just as we did in the win/lose imbalance bias. But I don’t believe it’s an actual problem.
