*Choccy Milk Testing Method (1k+ tests/hour)

[Gragg9]

Hey y’all! As many of you know I’ve spent a fair amount of time this season trying to figure out the sim. I found a LOT of ways *not* to sim test. At times it seems like the sim is designed not to be tested or figured out. Despite that a few people (maybe many people?) have figured out how to do some decent mass testing. I’m stubborn so I kept plugging away until I came up with what I think is a decent method. It's not complicated and probably what most war rooms are doing already. I’m a big proponent of making sim knowledge and sim testing methods public so here’s a quick guide to how I’ll be doing my testing going forward. I’m quite sure there are ways to improve it and would love to hear your ideas, questions, and concerns in the comments or in my dms. Without further ado, here’s how it goes.

---

What you’ll need:

A copy of the sim (if you do it quick it’s half off)

A macro recorder (I use https://www.macrorecorder.com/ but there are probably better)

Blank save with the league settings 

Spreadsheet to create duplicate rosters 

Most recent week’s save (to see what strats/depth your opponent used)

Spreadsheet to interpret the data

Note - Spreadsheets provided are examples that were hastily put together. I recommend making better ones if you can.

---

Setting it up

This is the annoying part of this method, as once it gets running it’s all automated. Meaning you can go do all of those chores you’ve been putting off while the sim runs 1,000 games. To set it up though you’ll need to launch the game and load up the blank save. This save has a custom schedule uploaded so that each team will only play one team throughout the season. They alternate home/away games to avoid any advantage. These teams carry the DSFL brandings but as they are completely blank they do not correlate to a DSFL team. We’ll be uploading rosters and changing strategies so they can be ISFL teams as well.

So now we have to get these custom rosters. Not gunna lie it's a bit of a process but luckily it’s all spreadsheet stuff. I have a rudimentary one for public use above but I highly recommend you force your team wizard to make a better one. Mine is set up using an import from the TPE tracker but I haven’t gotten it super stable yet.

The sheet should be self explanatory but since word count determines payout I’ll explain it here as well. Put the two teams you want to pit against each other in B1 and B2 (where the sheet tells you to). It uses the team abbreviations instead of full names (SAR, OCO, NOLA, BBB, etc). The rosters will be generated in column C along with the duplicates. This will look like a massive jumble of words but that's the format the game can read. From here, copy and paste the *entire* column C to notepad and save as a ‘.json’ file. This can be done by simply clicking save as and changing the extension to .json. To upload this file, go back to the sim and click ‘commissioner's office’ in the top right corner. There will be an ‘import players’ button. Make sure the ‘Place on Existing Teams’ option is checked and upload. You’ll now have 4 rosters of identical players for both teams you’re testing.

Despite all the words so far the process up to this point is very quick. Here’s where that annoying setup comes in. You’ll need to set the strategies for you and your opponent, along with depth charts manually. Now before you close out the guide, let me say that I only do this for 2 teams instead of all 8. This still returns 1000 tests per hour which I think is plenty and still saves a ton of testing time in the long run. If you’re very motivated and want to set up all 8 you can do that for 5k+ tests per hour, and test multiple strats at the same time. My process here is to pull up my team’s DC and input it. THEN SAVE IT, and load up the most recent weeks file. Grab a few screenshots of their strategies and depth charts. Instead of taking a picture of each formation I usually just note down the minor differences between them. It’s easy just to use the built in copy tool for formations and make a couple changes instead of inputting the depth charts over and over. I haven’t timed it but I think this brings it up to about 15 minutes of setup before testing can begin. If anyone has ideas for how to make this quicker, please let me know. I’m quite sure there’s a way that I’ll think of as soon as I publish. DON’T FORGET TO SAVE AFTER SETTING UP THE FILE.

Simming

Ok now we’re ready to test. Boot up your macro recorder (I think AHK works here but I know very little about it). You’ll want to sim the *whole season* (more on this later), Export Data (button in the top right), then reload the file. Then you’ll import the data to whatever spreadsheet you’re using and repeat the process. The files will probably be found in Documents>WolverineStudios>Leagues>’SaveName’. I usually use the _games file but take a look through each of them to decide what data you want.  The macro will remember all mouse/keyboard actions if you’ve set it to record, so you only have to do this once. From there the macro will repeat as many times as you want. On my computer this all takes 40 seconds for 14-56 games simmed depending on how much setup you’ve done. This is capable of getting to meaningful sample sizes quickly and easily as you can walk away to do chores and come back to 500 games simmed and their data saved to a spreadsheet.

Since I think clarity may be getting lost in the details:

---

Setup
Load blank Save (with cleared rosters)
Import duplicated rosters (from linked sheet or better one)
Set Strategies and Depth Charts (Manually :{ )
> Macro start
Sim Season
Export Data
Reload Save
Import Data to Spreadsheet
> Macro end
Analyze results, make changes, repeat macro

---

Note on Morale

Now one of the primary concerns about mass testing in the new sim has been morale. It changes for every player after every game. So if you were to sim an entire season, you’d have morale supposedly influencing those later games and skewing results. Being the nerd I am I decided to put this to the test and see exactly how much morale influenced results. To do this I used this save filled with 8 duplicate rosters with duplicate strats and depth charts. I then set 4 teams to max morale and the others to minimum morale. After nearly 1,000 games simmed there was no statistical significance to the win rate, points for, or points against one or the other. Nearly all other strategy or depth changes lead to a statistical significant difference by that sample size. Despite no change in win rates or scores, there were constant fights breaking out in locker rooms. This is highly entertaining and I strongly recommend we set everyone's morale to 1 in future seasons of the ISFL.

Here's the save I used to run the tests. Y’all may find it useful for testing other things as well.

There are several people out there that say morale is super important so if there’s something I’m not considering please let me know. I do have stupid moments so please don’t take my word for it.

---

If anyone has trouble or questions regarding any part of this guide please let me know. I can’t promise I’ll be super quick to respond but I’ll try to help and answer questions any way I can. Again if you have an idea about how to improve mass testing or find any problems with this one please let me know. Let’s make the sim suck less.

[shrub02]

Give this man the bonus

[Yeenoghu]

Now one of the primary concerns about mass testing in the new sim has been morale. It changes for every player after every game.

It degrades for the tester after each game too.
Great job! Always love seeing innovation like this with the sim.

[Number 82]

:eyes:

[katarn22]

[Kotasa]

The youngins don't know nothin, all those days trapped in the sim dungeon, no sunlight for days on end just to see the needle move .00002% and consider it a victory. On top of discovering it was all based on your system clock so our hours of testing were for NAUGHT!!!

[PMoney]

so our hours of testing were for NAUGHT!!!

Official vocab guidelines say that we should refer to this as 'for ROLY'

[Pat]

so our hours of testing were for NAUGHT!!!

Official vocab guidelines say that we should refer to this as 'for ROLY'

S Tier comment right here

[Gragg9]

Now one of the primary concerns about mass testing in the new sim has been morale. It changes for every player after every game.

It degrades for the tester after each game too.
Great job! Always love seeing innovation like this with the sim.

It took me a while to figure out how to get around this issue as well. After a while though I realized my morale bottoms out at 1! Not so bad!

[Jo3fish5]

CHOCCY MILK