Ranked Elo Overhaul

Leave your suggestions about the game here!

Re: Ranked Elo Overhaul

Postby Flake » Thu Feb 22, 2018 5:41 pm

Aviel wrote:I personally think that having 20 placement games would be too much, maybe 15 would do?

Otherwise I /support.

I agree 20 is slightly too many, though there is a large drawback in having <20 games; that is <20 games is a VERY small sample, especially for a team based game reliant on a multitude of factors outside of player skill. After 15 games, and even arguably after 20, you aren't really going to get an accurate indication of your actual skill at all.

I think you should be given an Elo that actually indicates your skill after your placement games, but at the same time I don't think you should have to play an excessive amount of games in order to get your Elo. It's extremely difficult to balance these two factors and therefore it's also difficult to decide the optimal amount of placement games, mainly due to the nature of the game.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Joacgroso » Thu Feb 22, 2018 6:10 pm

I think 20 is a good amount.
Sorry if I have spelling mistakes. English is not my main language.
I added a petition to my achievement rework!
Plese come and vote to support it!
If you have any suggestions related to achievements, please post them on my thread so we can suggest them together!
User avatar
Joacgroso
Arsonist
Arsonist
 
Posts: 2665
Joined: Mon May 02, 2016 6:21 pm
Location: Argentina

Re: Ranked Elo Overhaul

Postby Flake » Tue Feb 27, 2018 11:45 am

A new poll has been added, since I believe this Overhaul will not make many further changes, and if changes were to occur, they would be minor. PLEASE make sure that you have read ALL of the Elo Overhaul before voting.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby rickms » Thu Mar 01, 2018 6:43 pm

Hi All,

First, I wanted to give Flake and everyone who participated in this discussion a thank you for your hard work and suggestions. I'll be frank I have not had the time yet to go over this math and give it the due diligence it deserves.

Second,
I did want to address the perception that getting the highest rating is just a "Grind". Quite frankly the numbers we have don't show this is the case.

There are approximately 60 players in the Master tier.

[*] 17 of these people have over 1000 games played. These players are NOT clustered at the top, they are spread rather evenly up and down the top 60.
[*] Two people in the top 15 have less than 500 games played.
[*] Two people in the top 60 have less than 250 games played.
[*] One person has over 2000 games played and they are not the highest elo player (very close, but not quite).

That being said, numbers aren't everything. If the rating FEELS like a grind, one can argue that is just as bad as it being just a grind.

Third,

Examining win rate alone is not a fair way to assess the system. If you are a very good player you are going to probably fly through the lower ranks, gaining much more rating per win, then once you reach master. If at master you start losing a lot more, your rating will drop much slower than your win rate. Faction win rate IS factored into your rating change as well, so add that into the mix. The win rate alone doesn't tell you the story of how "good" a player is. Given the random nature of the game and the reliance on team mates, there is no really perfect algorithm to determine how good someone is at a game of Town Of Salem. So what we did was start with a baseline system (ELO) and grafted on some adjustments to better suit our game. Among those are a faction win rate adjustment and a tier based K factor so higher tiers don't get clobbered when they lose.

Over the first two seasons (once bugs and exploits were fixed :P <3 ) we've seen ranked rating shape up quite well. The spread of rating is pretty tight and things are pretty competitive at the top.

Let it be known that it will be unlikely that we well ever be "done" tweaking the rating system. After every season we're going to be looking at the numbers and listening to community feedback (like this post) to see how we can possibly make things better.

At this point in time it is highly unlikely we will completely gut the current algorithm for a new one. We will continue to tweak the current system, wether it be placement games required, K factor, or a new modifier all together. We want everyone to enjoy ranked as much as possible!

Thanks
Rick
rickms
Witch
Witch
 
Posts: 47
Joined: Mon Jan 11, 2016 1:08 pm

Re: Ranked Elo Overhaul

Postby Flake » Thu Mar 01, 2018 7:45 pm

rickms wrote:Hi All,

First, I wanted to give Flake and everyone who participated in this discussion a thank you for your hard work and suggestions. I'll be frank I have not had the time yet to go over this math and give it the due diligence it deserves.

Second,
I did want to address the perception that getting the highest rating is just a "Grind". Quite frankly the numbers we have don't show this is the case.

There are approximately 60 players in the Master tier.

[*] 17 of these people have over 1000 games played. These players are NOT clustered at the top, they are spread rather evenly up and down the top 60.
[*] Two people in the top 15 have less than 500 games played.
[*] Two people in the top 60 have less than 250 games played.
[*] One person has over 2000 games played and they are not the highest elo player (very close, but not quite).

That being said, numbers aren't everything. If the rating FEELS like a grind, one can argue that is just as bad as it being just a grind.

Third,

Examining win rate alone is not a fair way to assess the system. If you are a very good player you are going to probably fly through the lower ranks, gaining much more rating per win, then once you reach master. If at master you start losing a lot more, your rating will drop much slower than your win rate. Faction win rate IS factored into your rating change as well, so add that into the mix. The win rate alone doesn't tell you the story of how "good" a player is. Given the random nature of the game and the reliance on team mates, there is no really perfect algorithm to determine how good someone is at a game of Town Of Salem. So what we did was start with a baseline system (ELO) and grafted on some adjustments to better suit our game. Among those are a faction win rate adjustment and a tier based K factor so higher tiers don't get clobbered when they lose.

Over the first two seasons (once bugs and exploits were fixed :P <3 ) we've seen ranked rating shape up quite well. The spread of rating is pretty tight and things are pretty competitive at the top.

Let it be known that it will be unlikely that we well ever be "done" tweaking the rating system. After every season we're going to be looking at the numbers and listening to community feedback (like this post) to see how we can possibly make things better.

At this point in time it is highly unlikely we will completely gut the current algorithm for a new one. We will continue to tweak the current system, wether it be placement games required, K factor, or a new modifier all together. We want everyone to enjoy ranked as much as possible!

Thanks
Rick

Thanks so much for the reply!

I've talked to quite a few people (including high Elo players) regarding how they feel about the current rating system, and the overwhelming majority (if not all) have noted that the main issues are to do with the K-Factor system and the fact that the system is too grind based.

I feel the main issue with the K-Factor system is that it punishes higher tier players far too much, and the amount they earn winning games is far too low as compared to gains when you are not in these higher tiers. A base K-factor of 16, >1800 = 11, >2000 = 7 gives far too significant of a drop in Elo change for those above 1800 Elo and especially those above 2000 Elo (correct me if these K-Factors are wrong). More appropriate might be: Base K-Factor = 16, >1800 = 12 or 13, >2000 = 10 or 11.

Regarding the issue of grind, I do agree that having a higher amount of games should give you more Elo, but if and ONLY if you have the win rate to prove this. This rework does exactly that - if you have a consistently good win rate, over time you will gradually earn more and more still. But grind is NOT equal to the effect of gaining more Elo because of a consistently high win rate over a large amount of games. I don't think your statistics really prove the current system isn't grind based - in fact, I would argue it does the opposite. For example, TWO people in the top 15 having over 500 games played seems like an extremely low amount of players given the amount of games required. 500 games is a HUGE amount of games, especially since the hard reset was not too long ago, and the fact that 13 out of the 15 top Elo players have over 500 games played seems to prove the system IS grind based more than it isn't. I'd bet a VERY small proportion of the Ranked player base have 500 or more games played since the hard reset. It should be the case that it is relatively easy to get to the Master tier after around 150ish games MAXIMUM, but that simply is an almost impossible (or impossible) task for any player regardless of innate skill.

Yes, the system being ENTIRELY based on win rate wouldn't be fair (else you'd gain an infinite amount of Elo for winning your first game), but that isn't the case. My main point is that it should be the MAIN controlling factor (but not the ONLY controlling factor), and the Elo system should be more balanced around win rate than playing more games, but at the moment that is not the case. Players in Platinum (a high Elo tier) with a 35% win rate (significantly below the average Ranked player win rate) prove this.

It is completely understandable that the ENTIRE rework would not be implemented - I honestly just wanted some of the main issues with the current system to be highlighted while giving appropriate ideas as to how to fix these issues.

Thanks! :)
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Algus » Thu Mar 01, 2018 8:05 pm

rickms wrote:Hi All,

First, I wanted to give Flake and everyone who participated in this discussion a thank you for your hard work and suggestions. I'll be frank I have not had the time yet to go over this math and give it the due diligence it deserves.

Second,
I did want to address the perception that getting the highest rating is just a "Grind". Quite frankly the numbers we have don't show this is the case.

There are approximately 60 players in the Master tier.

[*] 17 of these people have over 1000 games played. These players are NOT clustered at the top, they are spread rather evenly up and down the top 60.
[*] Two people in the top 15 have less than 500 games played.
[*] Two people in the top 60 have less than 250 games played.
[*] One person has over 2000 games played and they are not the highest elo player (very close, but not quite).

That being said, numbers aren't everything. If the rating FEELS like a grind, one can argue that is just as bad as it being just a grind.

ranked as much as possible!

Thanks
Rick


I personally dislike the idea of any elo system that doesn't take every variable into consideration; the role, the individual player's winrate on that role (some are better at certain roles and worse at others) the actions they took on the role they played as; executing town for instance as jailor, whether or not your self-heal/vest saved you from an attack, how many successful heals you made, how long you saved/what you accomplished with your alerts, as well as the night you died for all roles of course. Early deaths are pretty much luck of the draw; although I think GF out of all evil roles should be hit the hardest for an early death, as it basically means they A) couldnt talk their way out of jail/off the stand early game or B) got ignited n2 (which I'll give a pass to)

Yea that's cluttered and some of this I believe you already have in place, though I have no proof.
(how the hell did I lose 7 elo as jester that got n2 vig shot?; - I assume because of the dead disguiser who visited me n1, in most cases that should be an easy win yes)

Also, the reason I quoted you here: who has the most games played, combined with the lowest winrate and what is their current elo? no need to name all I want is numbers.
IGN - Max
If I'm not on Max, I'm the Thunder Trainer with cupid/loveshack or wolfpup/wolfden
User avatar
Algus
Bodyguard
Bodyguard
 
Posts: 210
Joined: Sat Mar 05, 2016 12:47 am
Location: Indiana, the worst town in the state.

Re: Ranked Elo Overhaul

Postby Flake » Thu Mar 01, 2018 8:22 pm

Algus wrote:I personally dislike the idea of any elo system that doesn't take every variable into consideration; the role, the individual player's winrate on that role (some are better at certain roles and worse at others) the actions they took on the role they played as; executing town for instance as jailor, whether or not your self-heal/vest saved you from an attack, how many successful heals you made, how long you saved/what you accomplished with your alerts, as well as the night you died for all roles of course. Early deaths are pretty much luck of the draw; although I think GF out of all evil roles should be hit the hardest for an early death, as it basically means they A) couldnt talk their way out of jail/off the stand early game or B) got ignited n2 (which I'll give a pass to).

In game actions affecting Elo has been suggested many times, and it is simply far too difficult (or impossible) to implement without significant downsides.

Refer to viewtopic.php?f=14&t=81136 and viewtopic.php?f=14&t=79794#p2622389
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Algus » Thu Mar 01, 2018 9:21 pm

My objection to your suggested change is that it isn't by the individual win% of the role the player is on, or rather doesn't account for that. Faction winrates would have to be a part, yes, but individual winrates based on what role they get alone should be part of the final equation.

It has nothing to do with it not being action based, although having read those threads my opinion on that is the same. It's not impossible to find strictly good or strictly bad situations and to reward/punish them accordingly. Take witch/transported members of an action set out of the equation entirely since those would never be accurate; however little by little it's within the realm of possibility.

--Also, gaining elo should be a grind. To be perfectly honest, I'm fairly certain even if that weren't the case it would be in BMG's best interest if it were. So, either way really.

I also see no reason for you to start at 0 when what is it, 1000? is the standard or average ELO for just about any system using it.
It doesn't really matter where you start it; but why bother changing it? It's far more recognizable and more importantly it's arbitrary anyway, as the starting point doesn't actually matter. (leagues elo system, 1200-1499 is silver, 1500-1799 is gold, 1800-2099 is plat, 2100-2399 is diamond; past that you hit the masters tier; and despite them using an MMR system (as of maybe 5? 6? years ago), the old ELO system is still identical numbers wise, as several websites will show you.
IGN - Max
If I'm not on Max, I'm the Thunder Trainer with cupid/loveshack or wolfpup/wolfden
User avatar
Algus
Bodyguard
Bodyguard
 
Posts: 210
Joined: Sat Mar 05, 2016 12:47 am
Location: Indiana, the worst town in the state.

Re: Ranked Elo Overhaul

Postby Flake » Thu Mar 01, 2018 9:52 pm

Algus wrote:My objection to your suggested change is that it isn't by the individual win% of the role the player is on, or rather doesn't account for that. Faction winrates would have to be a part, yes, but individual winrates based on what role they get alone should be part of the final equation.

It has nothing to do with it not being action based, although having read those threads my opinion on that is the same. It's not impossible to find strictly good or strictly bad situations and to reward/punish them accordingly. Take witch/transported members of an action set out of the equation entirely since those would never be accurate; however little by little it's within the realm of possibility.

--Also, gaining elo should be a grind. To be perfectly honest, I'm fairly certain even if that weren't the case it would be in BMG's best interest if it were. So, either way really.

I also see no reason for you to start at 0 when what is it, 1000? is the standard or average ELO for just about any system using it.
It doesn't really matter where you start it; but why bother changing it? It's far more recognizable and more importantly it's arbitrary anyway, as the starting point doesn't actually matter. (leagues elo system, 1200-1499 is silver, 1500-1799 is gold, 1800-2099 is plat, 2100-2399 is diamond; past that you hit the masters tier; and despite them using an MMR system (as of maybe 5? 6? years ago), the old ELO system is still identical numbers wise, as several websites will show you.

Name a few actions in the game that are strictly and objectively ALWAYS entirely good or entirely bad. You will struggle. Also, my rework does already account for the win percentage of the player's faction. Accounting for the ROLE win percentage would make little or no difference in most cases, but I guess that could be added. I doubt you really understand what I mean by this - it would take a very long explanation to tell you why, but I will tell you if necessary. The matter of accounting for the ROLE win percentage is far more complex than you likely think it would be.

Gaining Elo SHOULD NOT be a grind. It shouldn't be the case that a player with 400 games is far more worse off than a player with 1000 games with the same win rate, as it is now. 400 games is more than a sufficient sample to showcase a player's true ability through their win rate, yet 1000 games with the same win rate would give you a considerably higher rating. That most definitely should not be the case. A player is hardly going to become significantly more skilled from 400 games played to 1000 games played, so why should their Elo significantly increase? It shouldn't. Elo should be a measure of skill, not experience, else you might as well rename it "Exp" instead of "Elo".

Starting at 0 makes more sense in terms of "less than 0 is below average, more than 0 is above average". You can do that because my rework is entirely balanced such that 0 would be the average rating of a given Ranked player. So, the reason is for ease of understanding. Conversely, I don't see why it would be mandatory for the rating system to be similar to other Elo systems. It is more recognisable, yes, but the system I have put in place is hardly difficult to comprehend. The starting point doesn't really matter anyway, as you say, it can be whatever the devs decide it to be.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Algus » Thu Mar 01, 2018 10:49 pm

experience does correlate to skill though. a few thousand games played here and playing a classic game for the first time in years, since I was playing with a friend who was new to ToS; I can tell you knowing exactly who mafia is as early as day 2 was extremely commonplace.

I use the same setup every game and after my first 20ish games in ranked this season, while i was still in silver and had the same people every match, the mafia actually started killing me n1 every. single. game. I finally got to the point (within that group of players) where theyd just protect me n1 because I was their carry. A medium or ret was all it took to solve the game outright in games I died n1, which is more or less why I climbed out. I haven't even played 200 this season. (or this rolelist period, I quit playing when they removed NB and guaranteed mafia "had the any" more or less)

A vig shooting an RM is literally impossible to be bad. Spin that any way you want to, It cannot be a bad thing. It confirms a vig, even worst case scenario (disguiser) it confirms the disguiser when the vig doesn't die to guilt a day later, which is strictly better than having no concrete verification. Do random shots exist? They do yes, so penalize non witched/transport/protected shots at town roles to compensate and there you go. (protected confirms a TP/RT slot at least, as well as the vig, again.)

Vig is by far the easiest however; and it may not be possible for every role to get these kinds of bonuses/penalties, but some roles can. If exploitation is a fear, cap it (ie a reward for successful doc heals, cap it at 3? would be the same as a vig or jailor killing 3 scum). I'm not saying add it to Doc or Jailor however, because I've seen doctors vehemently and misguidedly claim an NK was town because they had healed them etc. Jailors SHOULD hold on to the last killing role for mafia and find the RMs so I'll pass on that too.

I really don't care too much about the action based gain/loss but I have to assume they have some things in place already; that jester loss for instance. -7 ELO which can only be because the disguiser died by sk as a "jester". How else could you possibly get a -7 for an NE loss. The game gave me a perfect set up for a clean win, the vig n2'd me though so I guess he deserves some credit. Disguiser was the only dead guy n1, and that left me as one of two possible visits.
IGN - Max
If I'm not on Max, I'm the Thunder Trainer with cupid/loveshack or wolfpup/wolfden
User avatar
Algus
Bodyguard
Bodyguard
 
Posts: 210
Joined: Sat Mar 05, 2016 12:47 am
Location: Indiana, the worst town in the state.

Re: Ranked Elo Overhaul

Postby Flake » Thu Mar 01, 2018 11:10 pm

Algus wrote:experience does correlate to skill though. a few thousand games played here and playing a classic game for the first time in years, since I was playing with a friend who was new to ToS; I can tell you knowing exactly who mafia is as early as day 2 was extremely commonplace.

I use the same setup every game and after my first 20ish games in ranked this season, while i was still in silver and had the same people every match, the mafia actually started killing me n1 every. single. game. I finally got to the point (within that group of players) where theyd just protect me n1 because I was their carry. A medium or ret was all it took to solve the game outright in games I died n1, which is more or less why I climbed out. I haven't even played 200 this season. (or this rolelist period, I quit playing when they removed NB and guaranteed mafia "had the any" more or less)

A vig shooting an RM is literally impossible to be bad. Spin that any way you want to, It cannot be a bad thing. It confirms a vig, even worst case scenario (disguiser) it confirms the disguiser when the vig doesn't die to guilt a day later, which is strictly better than having no concrete verification. Do random shots exist? They do yes, so penalize non witched/transport/protected shots at town roles to compensate and there you go. (protected confirms a TP/RT slot at least, as well as the vig, again.)

Vig is by far the easiest however; and it may not be possible for every role to get these kinds of bonuses/penalties, but some roles can. If exploitation is a fear, cap it (ie a reward for successful doc heals, cap it at 3? would be the same as a vig or jailor killing 3 scum). I'm not saying add it to Doc or Jailor however, because I've seen doctors vehemently and misguidedly claim an NK was town because they had healed them etc. Jailors SHOULD hold on to the last killing role for mafia and find the RMs so I'll pass on that too.

I really don't care too much about the action based gain/loss but I have to assume they have some things in place already; that jester loss for instance. -7 ELO which can only be because the disguiser died by sk as a "jester". How else could you possibly get a -7 for an NE loss. The game gave me a perfect set up for a clean win, the vig n2'd me though so I guess he deserves some credit. Disguiser was the only dead guy n1, and that left me as one of two possible visits.

Yes, experience correlates to skill to some extent, but you're missing the point. Experience does not correlate to skill NEARLY AS MUCH AS a player's Win Rate correlates to skill. If a player can net a good win rate after a significant amount of games, then they are skilled at the game, because they perform better at the goal of the game, to win, better than others. Experience comes hand in hand with the fact that you will gain a significant amount of Elo after a large amount of games, GIVEN you have a high win rate. A player with 10 games and a 90% win rate will still not have as good of an Elo than a player with 100 games and a 90% win rate. Experience DOESN'T need to be accounted for through grinding, because it is already self-implied via increase/decrease in Elo due to having a larger sample of games with a given win rate.

A Vigilante shooting a Random Mafia in one instance does not mean that the Vigilante player had full knowledge or necessarily scum read the Random Mafia, but it could have been a random shot and the Vigilante got lucky by chance. Is rewarding random shooting a good thing? No. The Random Mafia could also gamethrow and out them self as being Random Mafia, and then the Vigilante could shoot them. Is the Vigilante playing well in this case? We see that the play of shooting a Random Mafia does not mean the player has played objectively good in all cases. Sure, it will have a good outcome on the game, but it does NOT mean that the good play occurred due to the player's skill. There are numerous factors affecting the objectivity of the scenario, including but not limited to the ones I have stated. You can see that each scenario becomes overly complicated, and finding which ones to implement (if any) would be nigh on impossible. It's far easier to simply use the win rate, which tells the whole story without the complexity.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Algus » Thu Mar 01, 2018 11:20 pm

i can name 2-3 players in masters right now with a winrate between 51-59% in ranked, you even said yourself town winrate/mafia winrates are roughly the same by themselves (less than 50%, both of them); Unless you intend to fix the rolelist as well you are equally misguided. Does a player getting carried or getting very lucky for a streak of games deserve to skyrocket in ranking? No. It should be consistent over hundreds of games. THAT proves skill, and that also CREATES skill where it doesn't inherently exist, because you learn a lot about behavioural patterns the more games you play.

By proxy of that alone you'll start winning more, when you deserve to, when you are certainly good at the game. Not a fast track.
IGN - Max
If I'm not on Max, I'm the Thunder Trainer with cupid/loveshack or wolfpup/wolfden
User avatar
Algus
Bodyguard
Bodyguard
 
Posts: 210
Joined: Sat Mar 05, 2016 12:47 am
Location: Indiana, the worst town in the state.

Re: Ranked Elo Overhaul

Postby Flake » Thu Mar 01, 2018 11:36 pm

Algus wrote:i can name 2-3 players in masters right now with a winrate between 51-59% in ranked, you even said yourself town winrate/mafia winrates are roughly the same by themselves (less than 50%, both of them); Unless you intend to fix the rolelist as well you are equally misguided.

Does a player getting carried or getting very lucky for a streak of games deserve to skyrocket in ranking? No. It should be consistent over hundreds of games. THAT proves skill, and that also CREATES skill where it doesn't inherently exist, because you learn a lot about behavioural patterns the more games you play.

By proxy of that alone you'll start winning more, when you deserve to, when you are certainly good at the game. Not a fast track.

I'm not responding to your first point because I don't really understand what you're trying to say. Yes, Master players can have a slightly above average win percentage as it is, I never said they can't. What are you trying to prove? Why does Town and Mafia having roughly equal win rates make me misguided?

There is a skyrocket in ranking with a lower amount of games even with the current system, though admittedly this rework does make more of an impact on earlier games than the current system. There is no real good way of measuring a player's skill with a lower amount of games, because the sample is simply too small. Make Elo changes too high, then you're giving RNG and "win streaks" too much of an impact as opposed to player skill. Make Elo changes too low, the player may not get the Elo they deserve given their amount of skill. It is very hard to balance the two factors, and this is honestly one of the things I struggled to balance when making this. K-factors influence this point entirely, and K-factors can be changed with ease, so it's not like this issue is extremely difficult to solve (if it needs solving).
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby rickms » Fri Mar 02, 2018 1:22 pm

You have to play enough games and maintain this winrate to get to masters. People have done it in as "little" as 250, while there are people with 700 games played and in Bronze. In this sense it might appear to be grindy, but enough games need to be played to make sure RNG is sufficiently factored out. If you can win a lot, over a long period of time (games) then you've shown you've earned that rating. If you've won on harder to win faction, then you're rating will be even higher.

The very valid question is, is this number of games (at least 250, with the current calculations and a very good win rate) too high? That is definitely something I'll bring up with the team.

As far as the other suggestion...

Factoring in every single decision one could make as each individual role is just not a viable path in this game. We have discussed modifiers based on individual role win rates, however those would probably vary dependent on the role list. So not only is the role important but the combination of roles are important. Having win rates for every combination of roles isn't really all that viable either.

Algus wrote:i can name 2-3 players in masters right now with a winrate between 51-59% in ranked


A vast majority of people in ranked have this win rate range. BTW.

Great feedback all. Keep it up!
rickms
Witch
Witch
 
Posts: 47
Joined: Mon Jan 11, 2016 1:08 pm

Re: Ranked Elo Overhaul

Postby Flake » Fri Mar 02, 2018 9:50 pm

rickms wrote:You have to play enough games and maintain this winrate to get to masters. People have done it in as "little" as 250, while there are people with 700 games played and in Bronze. In this sense it might appear to be grindy, but enough games need to be played to make sure RNG is sufficiently factored out. If you can win a lot, over a long period of time (games) then you've shown you've earned that rating. If you've won on harder to win faction, then you're rating will be even higher.

The very valid question is, is this number of games (at least 250, with the current calculations and a very good win rate) too high? That is definitely something I'll bring up with the team.

I don't think you need all that good of a win rate to achieve Master tier. Many people with a half decent win rate can simply climb by grinding games to get to Master, including myself, which shouldn't be the case. Obviously, in the early stages of the release, it looks like Master tier is very difficult to get into, because people have not yet played enough games. However, in a few months time, I can almost guarantee that Master tier will NOT be a Tier that has hardly anyone in, and after a lot of time it would be relatively commonplace as compared to the rarity it should have. I should also note that, as it is, once you enter Master tier it is almost impossible to leave it - this should not be the case. It should be a bit difficult to leave it, yes, but not IMPOSSIBLE. By "leaving it" I refer to going a relatively substantial amount under the cap for Master Elo. The percentages of players in each tier should consistently be the same over time, but at the moment it is clear that is NOT the case. Players in Master will continue to grind, and grind, and grind, until they hit 3000+ Elo, then continue on and continue going up, just like it was before, albeit far more slowly due to the extremely detrimental K-Factor for Master Tier. Percentage of players in each tier will slowly but surely be skewed towards the higher Elo Tiers due to grinding, just like it was before.

I would say getting to Master Tier in 250 games would be very difficult, and would require a VERY high Win Rate over these 250 games. Even if this is the case, 250 games most definitely should not be the "minimum" amount of games required to realistically enter the Master Tier. 150 or so should be more than a sufficient sample size of games in order to realistically enter Master Tier.

The K-Factor (involving determining the minimum realistic requirement of games to get into Ranked) should most definitely be your second priority AFTER removing grinding from the system ENTIRELY. Grinding has absolutely no benefits for the Elo system as a whole, as "experience" is already self implied in an increase of Elo after more games with a consistently good win rate, and vice-versa with a decrease in Elo.

TL;DR, everything might seem decent as it is because we cannot yet fully see the effects of slow grinding, but in the long term it will most definitely become a growing issue. The minimum realistic requirement of amount of games to be able to get into Master tier should most definitely be much lower, 100-150 is more than enough to be able to prove player skill sufficiently. 1st priority is removing grinding, 2nd priority is fixing K-Factors.

Again, thank you so so sooo much for reading and replying! Sorry if I come off as overly argumentative, I just like to get my points across. If you want to know anything else of my opinions on the matter or whatever, feel free to message me either here or in PMs.

Thanks! :)
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Algus » Sat Mar 03, 2018 1:53 am

What's the obsession with fast tracking people to higher ranks? I don't even want this.

Make them earn it. If your retort is "not everyone has the time to grind"

Mines going to be "not everyone deserves a higher ranking then"
IGN - Max
If I'm not on Max, I'm the Thunder Trainer with cupid/loveshack or wolfpup/wolfden
User avatar
Algus
Bodyguard
Bodyguard
 
Posts: 210
Joined: Sat Mar 05, 2016 12:47 am
Location: Indiana, the worst town in the state.

Re: Ranked Elo Overhaul

Postby Flake » Sat Mar 03, 2018 9:27 am

Algus wrote:What's the obsession with fast tracking people to higher ranks? I don't even want this.

Make them earn it. If your retort is "not everyone has the time to grind"

Mines going to be "not everyone deserves a higher ranking then"

It's not FAST TRACKING people to higher ranks. At the moment, it takes a BARE MINIMUM (realistically) of 250 games to get to Master Tier, when 100-150 games is more than sufficient. It's not fast tracking, it's just FASTER than it currently is, and the current speed is simply slower than it should be.

Your last statements are stupid. There will still be a sufficient amount of games the player has to play in order to get to higher ranks, as it should be, but there are simply less than the currently very high amount required. Also, I don't think you fully understand what "grind" means. Grind doesn't mean "the more games you play, the more you can potentially earn", it means "the more games you play, the more you can potentially earn PLUS free Elo for no reason". People still must take the time to get to a higher rank, simply not as long as it currently does. Believing that if "not everyone has the time to grind" they should not be able to get to a higher rank, not only is this point partially invalid in this case (100-150 games is still a lot of games to determine player skill), but it is also the wrong way to think about it - a sufficient sample size should be ALL that is needed to be able to tell a player's true skill. A player who is bad but "grinds a lot of games to make up for it!" should not be rewarded (as it currently is), and if you think this is incorrect then your opinion is greatly skewed.

Do you honestly think a 35% win rate player (bearing in mind 50% is around the average Ranked player win percentage) should be in PLATINUM? If you think this is fine, then you are flat out wrong. You shouldn't be rewarded for the amount of games you play, you should be rewarded for the amount of games you play GIVEN a consistently higher than average win rate. 35% after many games is not a consistently higher than average win rate, so why is this player getting rewarded? They are being rewarded simply because they "play many games" - do you think this should be the case?? I don't really see what's so hard to comprehend about this.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby rickms » Mon Mar 05, 2018 12:02 pm

Flake wrote:
The K-Factor (involving determining the minimum realistic requirement of games to get into Ranked) should most definitely be your second priority AFTER removing grinding from the system ENTIRELY. Grinding has absolutely no benefits for the Elo system as a whole, as "experience" is already self implied in an increase of Elo after more games with a consistently good win rate, and vice-versa with a decrease in Elo.



So we're on the same page, let's start with a definition.

Grinding, by my/our definition, is playing more. Period.

There is no evidence that the more you play, the higher rank you are. There is a correlation between those who are high ranked and a large number of games played, but it's important to understand that "correlation is not causation". Meaning just because the higher ranked players have a lot of games played, doesn't mean all you have to do is play a lot to get a higher rank. The data simply doesn't support that conclusion.

Playing more and winning at a higher rate than other people will get you to a higher rating. This is by design of any rating system. If you win more than you lose, you gain rank. The more you "win more than you lose", the more rank you gain, with the goal being you will eventually get to a rating where your talent equals that of the players around you, your win rate will drop to ~50% and you will no longer go up in rating.

Obviously, in the early stages of the release, it looks like Master tier is very difficult to get into, because people have not yet played enough games. However, in a few months time, I can almost guarantee that Master tier will NOT be a Tier that has hardly anyone in, and after a lot of time it would be relatively commonplace as compared to the rarity it should have.


The season is already 3 months old and probably won't last too much longer. After 3 months only 60 people reached master. That is VERY low. One could argue that there are too few people, not too many.

Thanks again!
Rick
rickms
Witch
Witch
 
Posts: 47
Joined: Mon Jan 11, 2016 1:08 pm

Re: Ranked Elo Overhaul

Postby Flake » Mon Mar 05, 2018 12:37 pm

rickms wrote:So we're on the same page, let's start with a definition.

Grinding, by my/our definition, is playing more. Period.

There is no evidence that the more you play, the higher rank you are. There is a correlation between those who are high ranked and a large number of games played, but it's important to understand that "correlation is not causation". Meaning just because the higher ranked players have a lot of games played, doesn't mean all you have to do is play a lot to get a higher rank. The data simply doesn't support that conclusion.

Playing more and winning at a higher rate than other people will get you to a higher rating. This is by design of any rating system. If you win more than you lose, you gain rank. The more you "win more than you lose", the more rank you gain, with the goal being you will eventually get to a rating where your talent equals that of the players around you, your win rate will drop to ~50% and you will no longer go up in rating.

Players that have a far below average win rate (~35%) can be placed in tiers far above average tiers (Platinum). That in and of itself means that amount of games played has FAR too much of an effect on Elo, and therefore this point alone is plenty of evidence to back up the hypothesis "the more you play, the higher rank you are".

I also fail to see how you can reliably draw ANY conclusion from the statistics you posted... I certainly can't. You can't simply say "oh, 2/15 people in Masters have above below 500 games played? Yeah that definitely means amount of games played doesn't support the idea of the more you play, the higher rank you are". No. Neither you nor I could ever draw a suitably objective conclusion from any of the statistics you posted, as it is entirely subjective because there is no baseline or "significance" test we can use to measure either of our hypotheses. The point I have stated above, however, IS objectively correct and DOES support my hypothesis.

Elo gain isn't entirely influenced by amount of games played, I am aware of that, but it doesn't change the fact that it is far too heavy of an influence at current. I fail to see why there is even a dispute of "whether grinding has too much or too little effect" at all, because grinding SHOULDN'T exist altogether. There should be absolutely no doubt whatsoever. Again, Elo should rise only with a higher amount of games played IF AND ONLY IF the player's win rate is also above average, and at the moment this is not always the case when it obviously should be. In a game like League of Legends, there would be uproar if said weaknesses were to occur, but in this game a far from good Elo system (albeit not awful) is "let off" simply because the game is more casual in general and has a smaller player base. Obviously, this should not be the case, and I'm sure you agree (I would hope).

rickms wrote:The season is already 3 months old and probably won't last too much longer. After 3 months only 60 people reached master. That is VERY low. One could argue that there are too few people, not too many.

Thanks again!
Rick

I'm not saying that REALISTICALLY many people would be able to get into Master tier (given the amount of time in each Season), nor did I say 60 people is high (I agree it is probably around the right amount or too low). What I am saying is that, if you WERE to leave the current system as it is running for say, 2 years, the weaknesses of the current system (both grinding and bad k-factor ratios) would become far more obvious than they already are, which is slightly scary given the fact that the effects of said weaknesses are already pretty prevalent.

Thanks!
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby rickms » Mon Mar 05, 2018 5:50 pm

Flake wrote:Players that have a far below average win rate (~35%) can be placed in tiers far above average tiers (Platinum). That in and of itself means that amount of games played has FAR too much of an effect on Elo, and therefore this point alone is plenty of evidence to back up the hypothesis "the more you play, the higher rank you are".


There is no one with a less than 40% win rate, currently in platinum.

I also fail to see how you can reliably draw ANY conclusion from the statistics you posted... I certainly can't. You can't simply say "oh, 2/15 people in Masters have above below 500 games played? Yeah that definitely means amount of games played doesn't support the idea of the more you play, the higher rank you are". No. Neither you nor I could ever draw a suitably objective conclusion from any of the statistics you posted, as it is entirely subjective because there is no baseline or "significance" test we can use to measure either of our hypotheses. The point I have stated above, however, IS objectively correct and DOES support my hypothesis.


What I posted was just examples of the data.

Your hypothesis: Ranked is grindy. AKA The more you play the higher you get in ranked.

We can test this by getting a sample of the people who have played the most, say the top 100 this season. Sort them by games played, and evaluate their current rating. If your theory were to hold, we should see a pattern of descending ratings that match game plays. Looking at the data, this is not the case. One guy near the top is in Gold.. there's even a Bronze player on this list, with a win rate of 35%, who's in the top 100 games played. If it were as simple as the more you play the higher you get, this guy shouldn't even be able to exist.


Elo should rise only with a higher amount of games played IF AND ONLY IF the player's win rate is also above average, and at the moment this is not always the case when it obviously should be.


This would be true, if we didn't factor in faction win rates to ELO calculation. All things being equal winning, playing against the same calibur players, 1 win + 1 loss should mean you have a net change of nearly 0. The rating calculation isn't that simple. If you win one game as an SK and then lose a game as town, against equally skilled opponents, you will rise in rating.


In a game like League of Legends, there would be uproar if said weaknesses were to occur, but in this game a far from good Elo system (albeit not awful) is "let off" simply because the game is more casual in general and has a smaller player base. Obviously, this should not be the case, and I'm sure you agree (I would hope).


Whether "ToS" is a causal game compared to LoL, I assure you, is not a factor at all into how we design our rating system. ToS has many intracies with roles, opponents, win rates that make it nearly impossible to impose a perfect rating system.

I'm not saying that REALISTICALLY many people would be able to get into Master tier (given the amount of time in each Season), nor did I say 60 people is high (I agree it is probably around the right amount or too low). What I am saying is that, if you WERE to leave the current system as it is running for say, 2 years, the weaknesses of the current system (both grinding and bad k-factor ratios) would become far more obvious than they already are, which is slightly scary given the fact that the effects of said weaknesses are already pretty prevalent.


This boils down to a philisophical question.

If you have played 200 games and won 60%, and I've played 400 games and won 60%, should we be the same rating?

If you answer is yes, then I ask,

If I go on a hot streak and play 200 games and win 75% is it the best experience for the Season and the other playing competative for me to quit the season there and not risk dropping my win rate? Is that fair to the others? What about the guy who played 500 games at 70%, going on that system alone he shouldn't be ranked higher... and then the more you play the harder it is to raise your win rate, by pure math. So in this case whoever gets on the best streak to start is the highest rated player?

Food for thought.

As always, even though we may not agree on points, all the feedback is welcome and shared (even if refuted!)
rickms
Witch
Witch
 
Posts: 47
Joined: Mon Jan 11, 2016 1:08 pm

Re: Ranked Elo Overhaul

Postby Flake » Mon Mar 05, 2018 8:16 pm

rickms wrote:There is no one with a less than 40% win rate, currently in platinum.

I apologise for this inaccuracy, my memory is apparently awful - I was almost certain I found someone on discord who fit the criteria, but I can't find it. I have however found a few cases where similar results can be found:

https://prnt.sc/indx0p - 42% and Master Tier
https://prnt.sc/indx2b - 39% and close to Platinum Tier

rickms wrote:Your hypothesis: Ranked is grindy. AKA The more you play the higher you get in ranked.

We can test this by getting a sample of the people who have played the most, say the top 100 this season. Sort them by games played, and evaluate their current rating. If your theory were to hold, we should see a pattern of descending ratings that match game plays. Looking at the data, this is not the case. One guy near the top is in Gold.. there's even a Bronze player on this list, with a win rate of 35%, who's in the top 100 games played. If it were as simple as the more you play the higher you get, this guy shouldn't even be able to exist.

Allow me to make it clear. My hypothesis is NOT "The more you play the higher you get in ranked", my hypothesis is "Grind EXISTS in the game to an extent, and to a larger extent than it should be EVEN IF grind should be in the game (which it shouldn't be)". My two screenshots above are more than enough evidence to prove this hypothesis correct, and I could get much more data to prove this.

rickms wrote:This would be true, if we didn't factor in faction win rates to ELO calculation. All things being equal winning, playing against the same calibur players, 1 win + 1 loss should mean you have a net change of nearly 0. The rating calculation isn't that simple. If you win one game as an SK and then lose a game as town, against equally skilled opponents, you will rise in rating.

Yes, but with more games, the effect of faction win rate becomes less and less. Thus, with a higher amount of games, the effects of faction win rate should be small (i.e with more games the percentage a player gets a non-beneficial faction will become closer and closer to the mean/average percentage across all players).

rickms wrote:Whether "ToS" is a causal game compared to LoL, I assure you, is not a factor at all into how we design our rating system. ToS has many intracies with roles, opponents, win rates that make it nearly impossible to impose a perfect rating system.

Fair point. A perfect rating system obviously is not achievable for a game like this, but games like LoL also have some complex interactions (albeit not nearly as much as Town of Salem) and they manage to maintain a close to perfect rating system.

rickms wrote:This boils down to a philisophical question.

If you have played 200 games and won 60%, and I've played 400 games and won 60%, should we be the same rating?

If you answer is yes, then I ask,

If I go on a hot streak and play 200 games and win 75% is it the best experience for the Season and the other playing competative for me to quit the season there and not risk dropping my win rate? Is that fair to the others? What about the guy who played 500 games at 70%, going on that system alone he shouldn't be ranked higher... and then the more you play the harder it is to raise your win rate, by pure math. So in this case whoever gets on the best streak to start is the highest rated player?

Food for thought.

As always, even though we may not agree on points, all the feedback is welcome and shared (even if refuted!)

This is actually an inherent issue with a good Elo system in any game, though perhaps not to the extent it would be in Town of Salem due to complex interactions that make win rate adjustment more sensitive than in other games. My idea to address this is a relatively simple one, though one that I think would prove to be VERY effective.

After a player's Elo, place a "significance value". This number measures how many games you have played and ONLY how many games you have played. In other words, it measures the significance of the Elo you have (open spoiler):

Spoiler: Significance Level       Games Required      Amount of Significance

Level (1)                      19 or less                  Negligible
Level (2)                      20 - 49                     Very weak
Level (3)                      50 - 99                     Weak
Level (4)                      100 - 149                 Moderate
Level (5)                      150 - 199                 Good
Level (6)                      200 - 249                 Strong
Level (7)                      250 - 349                 Strong
Level (8)                      350 - 449                 Strong
Level (9)                      450 - 599                 Strong
Level (10)                    600 - 799                 Very strong
Level (11)                    800 - 999                 Very strong
Level (12)                    1000 or more            Extremely strong
This type of system would be brilliant because this way you could easily distinguish between player Elo and Experience, and at the same time you can easily compare player Elo and Experience to see if the player's apparent "skill level (Elo)" is actually significant or not. Higher significance level means the person's Elo is more likely to be accurate. The player's significance level would be presented along with the player's Elo. For example, a player rated 1000 Elo with 650 games played will have an Elo of 1000 (10).

With this system, removing grind has next to no weaknesses. This system however could NOT be implemented if grind exists.

EDIT: Added suggestion to Original Post (Section 4). I have also adjusted Elo Tiers (Section 5) such that some Tiers require a specific Significance Level achieve said tier. In other words, your Elo alone doesn't determine your Tier now, your Significance Level also plays a part.

Thanks!
Last edited by Flake on Wed Mar 07, 2018 2:16 pm, edited 6 times in total.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Joacgroso » Tue Mar 06, 2018 11:14 am

Would the significance level be permanent, or would it be reseted at the start of each season.
Also, I assume it only takes in account ranked games played, right?
Sorry if I have spelling mistakes. English is not my main language.
I added a petition to my achievement rework!
Plese come and vote to support it!
If you have any suggestions related to achievements, please post them on my thread so we can suggest them together!
User avatar
Joacgroso
Arsonist
Arsonist
 
Posts: 2665
Joined: Mon May 02, 2016 6:21 pm
Location: Argentina

Re: Ranked Elo Overhaul

Postby Flake » Sun Mar 11, 2018 5:51 pm

Joacgroso wrote:Would the significance level be permanent, or would it be reseted at the start of each season.
Also, I assume it only takes in account ranked games played, right?

The Significance Level would be reset with a hard Elo reset, but with soft resets (or no reset) this would not be the case.

Yes, it is only for Ranked games played. All Ranked games from the last hard Elo reset would be counted.

I'll add these two points to OP - thanks!
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Flake » Sun Mar 25, 2018 2:20 pm

Bump.
Flake
Consort
Consort
 
Posts: 1074
Joined: Tue Oct 28, 2014 8:34 am
Location: England, UK

Re: Ranked Elo Overhaul

Postby Jackparrot » Fri Apr 13, 2018 2:22 pm

One thing i notice is how you can play with a wide varitety of skilled people EX: you are in silver, yet you can play with really low players(which drag down your ELO) or high players(which boosts your ELO). what i think about this is that the system should make you play within your rank (EX: silver) instead of three to show more skill.

rickms wrote:Hi All,

First, I wanted to give Flake and everyone who participated in this discussion a thank you for your hard work and suggestions. I'll be frank I have not had the time yet to go over this math and give it the due diligence it deserves.

Second,
I did want to address the perception that getting the highest rating is just a "Grind". Quite frankly the numbers we have don't show this is the case.

There are approximately 60 players in the Master tier.

[*] 17 of these people have over 1000 games played. These players are NOT clustered at the top, they are spread rather evenly up and down the top 60.
[*] Two people in the top 15 have less than 500 games played.
[*] Two people in the top 60 have less than 250 games played.
[*] One person has over 2000 games played and they are not the highest elo player (very close, but not quite).

That being said, numbers aren't everything. If the rating FEELS like a grind, one can argue that is just as bad as it being just a grind.

Third,

Examining win rate alone is not a fair way to assess the system. If you are a very good player you are going to probably fly through the lower ranks, gaining much more rating per win, then once you reach master. If at master you start losing a lot more, your rating will drop much slower than your win rate. Faction win rate IS factored into your rating change as well, so add that into the mix. The win rate alone doesn't tell you the story of how "good" a player is. Given the random nature of the game and the reliance on team mates, there is no really perfect algorithm to determine how good someone is at a game of Town Of Salem. So what we did was start with a baseline system (ELO) and grafted on some adjustments to better suit our game. Among those are a faction win rate adjustment and a tier based K factor so higher tiers don't get clobbered when they lose.

Over the first two seasons (once bugs and exploits were fixed :P <3 ) we've seen ranked rating shape up quite well. The spread of rating is pretty tight and things are pretty competitive at the top.

Let it be known that it will be unlikely that we well ever be "done" tweaking the rating system. After every season we're going to be looking at the numbers and listening to community feedback (like this post) to see how we can possibly make things better.

At this point in time it is highly unlikely we will completely gut the current algorithm for a new one. We will continue to tweak the current system, wether it be placement games required, K factor, or a new modifier all together. We want everyone to enjoy ranked as much as possible!

Thanks
Rick


You state that the players who have a lot of games are not at the top rickms, this is bad as it means that the most skilled players are not nesscarily distributed in the proper ELO.
I am not as active as I used to be, although I should be somewhat active.
Jackparrot
Sheriff
Sheriff
 
Posts: 579
Joined: Thu Mar 30, 2017 10:16 am

PreviousNext

Return to Suggestions

Who is online

Users browsing this forum: No registered users and 2 guests