Test Discussion Masterthread

Test roles that have been suggested in simulated gameplay.

Test Discussion Masterthread

Postby alex1234321 » Sat Feb 06, 2021 7:21 pm

I am making this thread so that we can discuss today's test. We can talk about it on Discord, but I'm also making a forum post so that it's permanent. That way, the devs and anyone looking back at this can see what happened during the test and how each of the tested roles played. The tested roles were the ones in the January Role Poll (Kamikaze, Framer buff, Ambusher buff, Consigliere Buff).

The first test was done in Discord and the second one was done in the client. Game 1 had 8 players and Game 2 had 7. There was a misgame in the client in between the two games, but I think the client was much easier to host. In the future, I think we should continue hosting tests in the old client. All four test roles appeared in at least one game.

Spreadsheet with actions from both games
Game 2 transcript

Overall, I think it was a successful first test. Unfortunately, we were not able to gather much information about the Framer, Consigliere, and Ambusher buffs since they were never used in either of the games. I think that suggests that the buffs don't substantially affect the roles, but more testing is needed to confirm that. Kamikaze was very powerful in the first game and was able to kill two Mafia members, ending the game for Town when it looked like scum would gain majority soon. This may suggest that the current version is overpowered, but like with the other roles one game isn't enough to confirm anything.

What did everyone else think? Feel free to discuss it regardless of whether or not you participated in the actual test.

Tests:
2/26/21 (Roles Tested: Consigliere Buff)
3/3/21 (Roles Tested: Rolestopper)
3/27/21 (Roles Tested: Copycat)
4/16/21 (Roles Tested: Stalker, Conqueror, Copycat, Rolestopper)
4/23/21 (Roles Tested: Spy Rework, Vigilante Rework, Mortician/Death)
4/30/21 (Roles Tested: Spy Rework, Vigilante Rework, Mortician/Death)
5/14/21 (Roles Tested: Spy Rework, Vigilante Rework, Mortician/Death)
5/28/21 (Roles Tested: Charmer, Technician, Kleptomaniac)
6/4/21(Roles Tested: Charmer, Agent, Technician, Kleptomaniac)
7/9/21 (Roles Tested: Confounder)
Last edited by alex1234321 on Fri Apr 23, 2021 8:38 pm, edited 6 times in total.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: 2/6/21 Test Discussion

Postby Brilliand » Sat Feb 06, 2021 7:39 pm

I mostly agree with your assessment, but I think it's important to consider that in that first game, a Vigilante would have been just as capable of saving the Town as that Kamikaze was.

The way the game actually played out: The Kamikaze and both Mafia died, leaving a single townie alive.

The way it would have gone with a Vigilante: One mafia dies, resulting in a 2v1 with the remaining Mafia known to all.
User avatar
Brilliand
Godfather
Godfather
 
Posts: 1503
Joined: Sat Nov 29, 2014 8:34 pm

Re: 2/6/21 Test Discussion

Postby alex1234321 » Sat Feb 06, 2021 8:03 pm

Brilliand wrote:I mostly agree with your assessment, but I think it's important to consider that in that first game, a Vigilante would have been just as capable of saving the Town as that Kamikaze was.

The way the game actually played out: The Kamikaze and both Mafia died, leaving a single townie alive.

The way it would have gone with a Vigilante: One mafia dies, resulting in a 2v1 with the remaining Mafia known to all.


Going into the night it was a 3v2. If there was a Vigilante instead, the vig would've killed maf and the Mafia would've killed someone else, making it a 2v1. In that situation, the Mafia member could've pushed a mislynch. If they got the vig, it would've been a win for Mafia and if they got the other player it would've been a draw. In the actual game the Mafia member was under suspicion, but if that weren't the case then a mislynch would have been possible. Also even if the second Mafia member wasn't under suspicion, the Kamikaze would still want to pick two targets since a 1f1 wouldn't change the balance of the game at all.

But overall I think it's fair for Kamikaze to have an advantage over Vigilante in those situations since it can't kill as many people.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: 2/6/21 Test Discussion

Postby Ezradekezra » Sat Feb 06, 2021 8:19 pm

Sorry I didn't end up showing up. I got dragged into a bunch of stuff at the last minute :/

Kamikaze sounds like it's a bit too OP in smaller games with fewer scum. Maybe we should hold off on testing it until we're able to get larger games.

Since none of the other roles were tested, I can't really say anything about them.
User avatar
Ezradekezra
Halloween 2020 Winner
Halloween 2020 Winner
 
Posts: 1296
Joined: Thu Jul 12, 2018 4:42 pm
Location: Whirl Islands, Johto Region

Re: 2/6/21 Test Discussion

Postby alex1234321 » Sat Feb 06, 2021 8:33 pm

Ezradekezra wrote:Sorry I didn't end up showing up. I got dragged into a bunch of stuff at the last minute :/

Kamikaze sounds like it's a bit too OP in smaller games with fewer scum. Maybe we should hold off on testing it until we're able to get larger games.

Since none of the other roles were tested, I can't really say anything about them.


I think we can continue testing it just to see how often it's able to get kills off and how successful it tends to be. Smaller games are similar to 15p endgames but with less information, so I think Kamikaze's performance in small games would be similar to but slightly worse than it would be in 15p games where it manages to survive into the endgame.

The other roles did each appear at least once, but their abilities weren't really used. Ambusher appeared in both games but never used cautious, which is interesting but might be due to having fewer players making immediate kills a higher priority. But that also means that in 15p endgames it probably won't be used which makes sense. Framer appeared in the second game but it took the tactical kill in the first night and was jailed in the second. Consigliere appeared in the first but only used the tactical kill. Interestingly, the Ambusher never took the kill if one of the other Mafia test roles was present, which means that those roles are probably still much weaker than Ambusher in endgames.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: 2/6/21 Test Discussion

Postby cob709 » Sat Feb 06, 2021 8:39 pm

To make a more accurate estimate, there must be a larger sample size. This would implicate 2-5 games of each role for accurate results.

Testing should be continued throughout February. Hopefully we pick up some more players for a 15p game.
I SEE ALL
User avatar
cob709
Mayor
Mayor
 
Posts: 874
Joined: Sat Aug 12, 2017 9:44 am

Re: 2/6/21 Test Discussion

Postby Brilliand » Sat Feb 06, 2021 9:17 pm

alex1234321 wrote:Going into the night it was a 3v2. If there was a Vigilante instead, the vig would've killed maf and the Mafia would've killed someone else, making it a 2v1. In that situation, the Mafia member could've pushed a mislynch. If they got the vig, it would've been a win for Mafia and if they got the other player it would've been a draw. In the actual game the Mafia member was under suspicion, but if that weren't the case then a mislynch would have been possible. Also even if the second Mafia member wasn't under suspicion, the Kamikaze would still want to pick two targets since a 1f1 wouldn't change the balance of the game at all.


It's also significant that the current version of the Kamikaze "wastes" the Mafia kill if they attack him the same night he shoots. In the actual game, the Mafia would have won outright if they had attacked the TK, and that TK turned out to be a Vigilante (while still sending the Ambusher to me). The Kamikaze's mechanics left no room for that play, however.
User avatar
Brilliand
Godfather
Godfather
 
Posts: 1503
Joined: Sat Nov 29, 2014 8:34 pm

Re: 2/6/21 Test Discussion

Postby Cavespider17 » Sun Feb 07, 2021 7:41 pm

Ezradekezra wrote:Sorry I didn't end up showing up. I got dragged into a bunch of stuff at the last minute :/

Kamikaze sounds like it's a bit too OP in smaller games with fewer scum. Maybe we should hold off on testing it until we're able to get larger games.

Since none of the other roles were tested, I can't really say anything about them.


Agreed on this - as a mafia memeber who was quashed - however, I think if we can get some more people - enough for a full match it will be fine.
User avatar
Cavespider17
Witch
Witch
 
Posts: 47
Joined: Wed Apr 22, 2020 9:19 am
Location: Europe

Re: 2/6/21 Test Discussion

Postby CapWarrior2 » Sun Feb 07, 2021 9:02 pm

Cavespider17 wrote:
Ezradekezra wrote:Sorry I didn't end up showing up. I got dragged into a bunch of stuff at the last minute :/

Kamikaze sounds like it's a bit too OP in smaller games with fewer scum. Maybe we should hold off on testing it until we're able to get larger games.

Since none of the other roles were tested, I can't really say anything about them.


Agreed on this - as a mafia memeber who was quashed - however, I think if we can get some more people - enough for a full match it will be fine.


Yea especially since atleast some mafia will live if there are more players, with us we just got obliterated. Vigilante still probably won town the game, but it wouldve been a 2v1, which could be still winnable. The first test was very fun and chaotic though.
Idea Index:
Click Here

Reworked Idea:
Omen - Neutral(Evil)

Testing Grounds:
Come visit!
User avatar
CapWarrior2
Doctor
Doctor
 
Posts: 187
Joined: Wed Sep 27, 2017 11:30 pm
Location: Somewhere with my Goat

Re: 2/6/21 Test Discussion

Postby alex1234321 » Mon Feb 08, 2021 7:25 am

CapWarrior2 wrote:
Cavespider17 wrote:
Ezradekezra wrote:Sorry I didn't end up showing up. I got dragged into a bunch of stuff at the last minute :/

Kamikaze sounds like it's a bit too OP in smaller games with fewer scum. Maybe we should hold off on testing it until we're able to get larger games.

Since none of the other roles were tested, I can't really say anything about them.


Agreed on this - as a mafia memeber who was quashed - however, I think if we can get some more people - enough for a full match it will be fine.


Yea especially since atleast some mafia will live if there are more players, with us we just got obliterated. Vigilante still probably won town the game, but it wouldve been a 2v1, which could be still winnable. The first test was very fun and chaotic though.

When Martin sent their action in to kill both scum I had to double check to make sure I didn't reveal the identity of the Mafia or something like that. I was expecting an easy scum victory at that point.

One issue with Kamikaze imo is that it there isn't much middle ground in how it performs. Assuming it uses its ability, it can either kill two scum or two Town. Sometimes it might get a 1f1 or a 1f2 but I'm not sure how common that would be, which we need more testing to determine. With Vigilante the most common outcomes are the more moderate ones, such as hitting one scum. However, it is fairly common for Vigilante to kill two Townies by missing its first shot, but this can be fixed fairly easily by giving it Jailor guilt. Removing one of its bullets to compensate could further reduce extreme outcomes. I'm not sure how we can balance Kamikaze to limit extreme results, but I'm open to ideas.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: 2/6/21 Test Discussion

Postby Cavespider17 » Mon Feb 08, 2021 9:44 am

alex1234321 wrote:
Yea especially since atleast some mafia will live if there are more players, with us we just got obliterated. Vigilante still probably won town the game, but it wouldve been a 2v1, which could be still winnable. The first test was very fun and chaotic though.

When Martin sent their action in to kill both scum I had to double check to make sure I didn't reveal the identity of the Mafia or something like that. I was expecting an easy scum victory at that point.

One issue with Kamikaze imo is that it there isn't much middle ground in how it performs. Assuming it uses its ability, it can either kill two scum or two Town. Sometimes it might get a 1f1 or a 1f2 but I'm not sure how common that would be, which we need more testing to determine. With Vigilante the most common outcomes are the more moderate ones, such as hitting one scum. However, it is fairly common for Vigilante to kill two Townies by missing its first shot, but this can be fixed fairly easily by giving it Jailor guilt. Removing one of its bullets to compensate could further reduce extreme outcomes. I'm not sure how we can balance Kamikaze to limit extreme results, but I'm open to ideas.[/quote]

According to the card if it kills a townie first the second attack doesnt go through - however, i think it should be unique for sure (if it isn't already).
User avatar
Cavespider17
Witch
Witch
 
Posts: 47
Joined: Wed Apr 22, 2020 9:19 am
Location: Europe

Re: 2/6/21 Test Discussion

Postby alex1234321 » Mon Feb 08, 2021 4:33 pm

Cavespider17 wrote:
alex1234321 wrote:
Yea especially since atleast some mafia will live if there are more players, with us we just got obliterated. Vigilante still probably won town the game, but it wouldve been a 2v1, which could be still winnable. The first test was very fun and chaotic though.

When Martin sent their action in to kill both scum I had to double check to make sure I didn't reveal the identity of the Mafia or something like that. I was expecting an easy scum victory at that point.

One issue with Kamikaze imo is that it there isn't much middle ground in how it performs. Assuming it uses its ability, it can either kill two scum or two Town. Sometimes it might get a 1f1 or a 1f2 but I'm not sure how common that would be, which we need more testing to determine. With Vigilante the most common outcomes are the more moderate ones, such as hitting one scum. However, it is fairly common for Vigilante to kill two Townies by missing its first shot, but this can be fixed fairly easily by giving it Jailor guilt. Removing one of its bullets to compensate could further reduce extreme outcomes. I'm not sure how we can balance Kamikaze to limit extreme results, but I'm open to ideas.


According to the card if it kills a townie first the second attack doesnt go through - however, i think it should be unique for sure (if it isn't already).[/quote]

Right now it's not unique. I'm hesitant to make it unique because right now it might be one of the best fakeclaims that scum roles have, and it's probably the only TK that scum can easily claim (since Veteran is unique and Vigilante can be witched). We probably need some large tests where there are multiple Kamikazes to see if it's too chaotic.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: 2/6/21 Test Discussion

Postby cob709 » Mon Feb 08, 2021 9:18 pm

btw tmk was good
I SEE ALL
User avatar
cob709
Mayor
Mayor
 
Posts: 874
Joined: Sat Aug 12, 2017 9:44 am

Re: 2/6/21 Test Discussion

Postby Brilliand » Mon Feb 08, 2021 9:37 pm

cob709 wrote:btw tmk was good


Yeah, that's a known-good idea, but all the same it's better if we can get away with not using it for these tests because the tests are more useful if we can stay true to the current state of Town of Salem as much as possible.

(Though I can see a counterargument where maybe the devs should prioritize TMK over any of the new ideas we test here, so maybe any ideas that advance from here to the main game will be added to an environment with TMK anyway.)
User avatar
Brilliand
Godfather
Godfather
 
Posts: 1503
Joined: Sat Nov 29, 2014 8:34 pm

Re: 2/6/21 Test Discussion

Postby cob709 » Tue Feb 09, 2021 12:08 am

Brilliand wrote:
cob709 wrote:btw tmk was good


Yeah, that's a known-good idea, but all the same it's better if we can get away with not using it for these tests because the tests are more useful if we can stay true to the current state of Town of Salem as much as possible.

(Though I can see a counterargument where maybe the devs should prioritize TMK over any of the new ideas we test here, so maybe any ideas that advance from here to the main game will be added to an environment with TMK anyway.)

nonononono

we need to treat these as actualy games and not tests, else we will receive biased results
if players act in a way just to satisfy the experiment, it will overall affect the outcome and the game and hurt the credibility of the results
I SEE ALL
User avatar
cob709
Mayor
Mayor
 
Posts: 874
Joined: Sat Aug 12, 2017 9:44 am

Re: 2/6/21 Test Discussion

Postby Brilliand » Tue Feb 09, 2021 12:56 am

cob709 wrote:we need to treat these as actualy games and not tests, else we will receive biased results
if players act in a way just to satisfy the experiment, it will overall affect the outcome and the game and hurt the credibility of the results


That's true, but it doesn't contradict what I said.

TMK isn't a player behavior thing, it's a game rules thing. It's either in effect ("using it") or not in effect ("not using it") for a given game.

I definitely didn't mean to imply that players in a TMK game should refrain from taking advantage of that rule for their own benefit!
User avatar
Brilliand
Godfather
Godfather
 
Posts: 1503
Joined: Sat Nov 29, 2014 8:34 pm

Re: Test Discussion Masterthread

Postby alex1234321 » Fri Feb 26, 2021 7:28 pm

I am making this into a masterthread where we can discuss each test afterward. Anyway we got 8 players for today's test, but the last Mafia member had to leave midgame so we stopped the test. After that we didn't have enough people to run another game. We had some very fun minigames though.

We weren't able to get too much info about any particular roles, but both the Consigliere and Framer rolled. The Framer took the kill N1 but was lynched the next day. During one of the nights, the Consig chose to investigate someone instead of killing, which suggests that the buff is very significant. The Consig found that there were two Bodyguards, which could be very useful in determining whether or not it is safe to kill the Jailor. While this wasn't a perfect test, it suggests that the Consig buff is very strong, which is a good thing.

What did everyone else think of the test?

Spreadsheet
Transcript
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: Test Discussion Masterthread

Postby Ezradekezra » Fri Feb 26, 2021 7:35 pm

I had fun. I can't comment much on the Consig buff, since I wasn't maf, though.

Speaking of those minigames we played, Stalker Fuckery was pretty fun, and I'd like to try it again to see if the Stalker manages to live long enough for things to get spicy.
User avatar
Ezradekezra
Halloween 2020 Winner
Halloween 2020 Winner
 
Posts: 1296
Joined: Thu Jul 12, 2018 4:42 pm
Location: Whirl Islands, Johto Region

Re: Test Discussion Masterthread

Postby alex1234321 » Fri Feb 26, 2021 7:51 pm

Ezradekezra wrote:I had fun. I can't comment much on the Consig buff, since I wasn't maf, though.

Speaking of those minigames we played, Stalker Fuckery was pretty fun, and I'd like to try it again to see if the Stalker manages to live long enough for things to get spicy.


It was fun being Jester in minesweeper. But I'd like to see what would happen if the Stalker isn't lynched D2.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: Test Discussion Masterthread

Postby alex1234321 » Wed Mar 03, 2021 12:18 pm

What do you guys think we should do about each of the roles that were tested in February? Should they be scrapped, approved, or tested further? Here are my thoughts:

Kamikaze: I hate to say this, but I think it should be removed based on our testing results. Even though it might just be OP in small games, it can completely swing the game if it makes it to the endgame. Also it involves at least two and usually 3 people dying, which is a lot for one night, especially for a Town role.

Ambusher Rework: We were not able to get very much information about it during the tests, probably because it is more important earlier in the game. Still I think we should test it 1-2 more times, particularly in larger games if possible. It seems like a good idea to me and I can't think of any gamebreaking consequences of the change.

Framer Rework: Same as Ambusher

Consigliere Rework: I think this change should be approved based on the 2/26 test. It gives a lot of info but it is definitely useful in helping with PoE and prevents the Mafia from targeting a protected Jailor because one TP is dead.

We probably won't have multiple large tests per week in the foreseeable future, so even if we can get 1-2 balanced tests of a role we should try to make as many conclusions as possible based on that information.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: Test Discussion Masterthread

Postby Ezradekezra » Wed Mar 03, 2021 8:04 pm

Spreadsheet from the games I was present for today

I got Brilliand to send me the transcript for the actual test game (game 1) so I could finish the spreadsheet.

The first sheet on there is a RNG list that I used when randing roles.

I think that the Rolestopper did pretty good, but they still got steamrolled because the Juggernaut was not quite as good. I wanna try running more tests with it in the future.
User avatar
Ezradekezra
Halloween 2020 Winner
Halloween 2020 Winner
 
Posts: 1296
Joined: Thu Jul 12, 2018 4:42 pm
Location: Whirl Islands, Johto Region

Re: Test Discussion Masterthread

Postby cob709 » Wed Mar 03, 2021 8:07 pm

Ezradekezra wrote:.

I think that the Rolestopper did pretty good, but they still got steamrolled because the Juggernaut was not quite as good. I wanna try running more tests with it in the future.

Agreed. In a real game, the rolestopper would've stopped the Lookout from seeing the Doctor or other TPs.
I SEE ALL
User avatar
cob709
Mayor
Mayor
 
Posts: 874
Joined: Sat Aug 12, 2017 9:44 am

Re: Test Discussion Masterthread

Postby alex1234321 » Wed Mar 03, 2021 8:08 pm

Here's a transcript transcript of the test:

I think Rolestopper was fairly effective in preventing and weakening Jailor meta. As Trapper, I didn't really change my strategy based on Rolestopper but probably would have if I was thinking about the role being in the game. I would change it to give roleblock notifications so that it isn't obvious that there's a Rolestopper.
Last edited by alex1234321 on Wed Mar 03, 2021 8:18 pm, edited 1 time in total.
#SaveTheTG

Tired of trying to play discord Mafia games and not getting enough people? Join Town of Morons! We now have our own bot!


Credit to PurpleSidewalk1
User avatar
alex1234321
Role Ideas Moderator
Role Ideas Moderator
 
Posts: 4508
Joined: Sat Nov 28, 2015 4:20 pm
Location: Somewhere (UTC-5)

Re: Test Discussion Masterthread

Postby Ezradekezra » Wed Mar 03, 2021 8:11 pm

What's everyone's opinion on letting Coven roles spawn, like for the test game we ran earlier?
User avatar
Ezradekezra
Halloween 2020 Winner
Halloween 2020 Winner
 
Posts: 1296
Joined: Thu Jul 12, 2018 4:42 pm
Location: Whirl Islands, Johto Region

Re: Test Discussion Masterthread

Postby Brilliand » Wed Mar 03, 2021 8:19 pm

Ezradekezra wrote:What's everyone's opinion on letting Coven roles spawn, like for the test game we ran earlier?


Thing is, Mafia and Town roles that get tested are mainly being tested for the purpose of Ranked, where Coven roles can't spawn. Spawning Stalker and Crusader in the same test game isn't great.

On the other hand, Conqueror was made with All Any in mind, and it's pretty hard to argue that CAA is worse than regular AA.

Seems to me that for this test month, we should switch back and forth between allowing Coven roles and disallowing Coven roles - and that answer may change in other months depending on the which roles are being tested. (In February, we had only Town and Mafia roles under test, so there was no reason to allow Coven at all.)
User avatar
Brilliand
Godfather
Godfather
 
Posts: 1503
Joined: Sat Nov 29, 2014 8:34 pm

Next

Return to Testing Grounds

Who is online

Users browsing this forum: No registered users and 9 guests