07-08-2020, 05:08 PM
(This post was last modified: 07-10-2020, 12:28 PM by Fordhammer.)
With some statements lately I thought it would be interesting to look deeper into the topic of sim testing. This is one of the important tasks to prepare for the upcoming matchup and to know what you can expect from a game. Every team - at least I guess so - tries it’s hardest to push its win percentage as high as possible for the given matchup. But who is best at doing so? Who knows. I try to answer this question in this article.
The simple analysis I do is based off the article I posted yesterday. There, I did a player evaluation on performance. I had to sim all the regular season games to gather the data I used there. With this work in mind, I already extracted the basic numbers (win%, pts scored, etc) for each game. To have a base I can compare to, I simmed all the regular season games again - but this time with the sim file from the prior week. As this is the base from what you start off for the next game (I assume?), this is the perfect number to compare the final numbers to. So upon my already 33k sim runs, I did another 30k to have a total of 63k sim-runs for this piece of media.
I know, the win probability calculated from 500 sim-runs per game is somewhat variable, and the average points scored are too. But I just use this numbers, to have something I can compare. At first, I show you all the numbers I gathered for each gameday, in the end, I’ll rank the teams based off of this numbers to crown the best sim testers in the league. Full stop.
Let’s start into the analysis. I have no numbers for gameday 1 to compare them to, as the preseason files are full of crappy strategies. So I’ll just post the win probabilities for every matchup and the final outcome. I will just post the win probability of the home team, as it’s not that hard to get the one from the away team, and it saves some numbers. I also don’t print the points scored, as this would be too much I assume. Beginning with week 2, I post the final win probability and the difference from the base probability for the home team.
Week 1
Yellowknife @ Baltimore 23-24, 38.1%
Philadelphia @ Colorado 17-19, 75.4%
Sarasota @ Chicago 20-9, 41.9%
Orange County @ Arizona 33-23, 42.3%
New Orleans @ San Jose 21-30, 37.7%
Honolulu @ Austin 17-37, 81.6%
Week 2
Colorado @ Baltimore 33-27, 44.7%, +1.8%
Sarasota @ Yellowknife 6-17, 80.2%, -1.2%
Chicago @ Philadelphia 19-17, 85.2, +0%
Honolulu @ Orange County 6-41, 82.0%, -1.8%
Arizona @ San Jose 24-13, 35.1%, +2.0%
Austin @ New Orleans 16-45, 61.5%, +11.0%
Week 3
Chicago @ Yellowknife 0-54, 90.2%, -2.4%
Baltimore @ Liberty 17-51, 66.3%, -0.6%
Colorado @ Sarasota 24-29, 48.9%, -3.0%
New Orleans @ Arizona 20-17, 60.1%, +5.6%
Orange County @ Austin 31-24, 64.9%, +11.6%
San Jose @ Honolulu 7-35, 75.8%, +2.8%
Week 4
Yellowknife @ Colorado 24-16, 62.3%, -2.0%
Baltimore @ Chicago 16-3, 37.9%, -7.4%
Philadelphia @ Sarasota 24-23, 60.7%, -0.2%
San Jose @ Orange County 20-17, 91.0%, +0.4%
Honolulu @ New Orleans 21-24, 83.0%, +5.2%
Arizona @ Austin 29-33, 74.1%, +3.6%
Week 5
Sarasota @ Baltimore 7-40, 60.9%, +3.6%
Philadelphia @ Yellowknife 23-24, 73.5%, +2.4%
Colorado @ Chicago 20-3, 30.7%, +2.4%
New Orleans @ Orange County 7-27, 69.7%, -4.8%
Austin @ San Jose 35-12, 26.7%, -6.2%
Arizona @ Honolulu 36-12, 54.3%, -0.4%
Week 6
Orange County @ Yellowknife 33-30, 54.9%, -4.8%
San Jose @ Colorado 10-37, 83.2%, -4.8%
New Orleans @ Philadelphia, 21-24, 51.7%, -1.0%
Chicago @ Austin 16-30, 87.4%, -2.8%
Baltimore @ Arizona 14-21, 66.5%, -0.8%
Honolulu @ Sarasota 17-23, 69.5%, +0.2%
Week 7
Orange County @ Baltimore 13-33, 44.7%, +5.0%
Honolulu @ Chicago 35-9, 41.3%, -5.2%
Sarasota @ Arizona 7-31, 63.1%, -5.6%
Yellowknife @ San Jose 34-27, 33.1%, +0.0%
Colorado @ New Orleans 24-26, 63.1%, -0.4%
Philadelphia @ Austin 17-7, 76.4%, +2.6%
Week 8
San Jose @ Baltimore 16-6, 68.1%, +1.8%
New Orleans @ Yellowknife 23-17, 68.3%, +3.8%
Austin @ Colorado 42-16, 64.5%, +0.6%
Arizona @ Chicago 17-22, 43.7%, +4.6%
Sarasota @ Orange County 20-24, 68.5%, -8.4%
Philadelphia @ Honolulu 12-21, 61.9%, -5.2%
Week 9
Arizona @ Philadelphia 34-13, 57.7%, -1.0%
San Jose @ Sarasota 14-20, 80.4%, -1.2%
Chicago @ Orange County 13-34, 89.4%, +0.8%
Baltimore @ New Orleans 13-25, 79.4%, +1.0%
Yellowknife @ Austin 23-17, 61.1%, +4.8%
Colorado @ Honolulu 3-6, 55.3%, -1.2%
Week 10
Austin @ Baltimore 20-24, 46.5%, -2.2%
Honolulu @ Yellowknife 14-40, 77.6%, +1.8%
Arizona @ Colorado 16-27, 66.7%, -6.4%
Orange County @ Philadelphia 13-24, 47.3%, +1.6%
Chicago @ San Jose 6-16, 69.5%, +0.2%
Sarasota @ New Orleans 24-30, 62.1%, -4.2%
Week 11
New Orleans @ Chicago 6-34, 41.7%, +4.2%
Austin @ Sarasota 13-14, 61.3%, +5.0%
Yellowknife @ Arizona 13-46, 52.3%, -0.2%
Colorado @ Orange County 15-9, 76.0%, +5.4%
Philadelphia @ San Jose 27-9, 47.3%, -10.8%
Baltimore @ Honolulu 14-10, 67.5%, +1.4%
So this are the final scores, the win probabilities for the games and the difference the home team had in win probability compared to the base win probability. There were some big changes in testing for some games, some are pretty close or even the same probability as the base was. Differences < 2% may also be just the fluctuation of the sim, but it’s hard to say. As already stated, I just used the numbers I got as given, and calculate with them the final sim testing rankings.
The best sim testers in the league…
full stop
are the guys of...
SARASOTA SAILFISH
Congrats.
Here is the final ranking
The Sailfish finish just close before Baltimore who also do an amazing job. Sarasota is quite average in its home games, but are taking their job super serious at away games it seems. Baltimore is the most balanced teams, they always seem to find a few percentage to boost their chances. Austin does amazing at home games, but should overthink what they are doing for away games. Philadelphia is similar to Sarasota, but only a bit less successful for both home and away games. New Orleans is on the same train as Austin, but can’t find the enormous success Austin is finding for their home games. Chicago is the last team who improves their win probabilities from the baseline. Honolulu, Yellowknife and Arizona all are pretty close at even, just a little below the 0. The three teams that are left, are all pretty bad in improving their win probability (maybe my sim runs did all just not go into their favor). San Jose at least is improving in away games, but struggles at home. Colorado isn’t really doing well in either, that’s why they are that low. The most surprising thing in that ranking for me was the last position. I didn’t expect the Orange County Otters being last in this ranking. I honestly expected them to battle for the #1 spot in this rankings. But they had some games were their win probability sunk far below the baseline. That sums up and find them only in the last spot here.
tl&dr
Sarasota, maybe: First, we don't have an entirely sim testing department. Second, by the time we play you our "WR who switched to QB" will be max passing stats and be as useful as young QB Cue. Third, when you actually test win percentage in the sim its a super low correlation between changing strategies and effectiveness in winning. So uh, I'm fine with not having a sim testing department. Fourth, I admire you attempting to bash a team's strategy for "doing sim testing" but if you're going to bash a team's strategy, it should be any team but Sarasota. I'm the best sim tester in the league, full stop. If we're running something its because its the optimal strategy for our team. But I mean, go off.
The simple analysis I do is based off the article I posted yesterday. There, I did a player evaluation on performance. I had to sim all the regular season games to gather the data I used there. With this work in mind, I already extracted the basic numbers (win%, pts scored, etc) for each game. To have a base I can compare to, I simmed all the regular season games again - but this time with the sim file from the prior week. As this is the base from what you start off for the next game (I assume?), this is the perfect number to compare the final numbers to. So upon my already 33k sim runs, I did another 30k to have a total of 63k sim-runs for this piece of media.
I know, the win probability calculated from 500 sim-runs per game is somewhat variable, and the average points scored are too. But I just use this numbers, to have something I can compare. At first, I show you all the numbers I gathered for each gameday, in the end, I’ll rank the teams based off of this numbers to crown the best sim testers in the league. Full stop.
Let’s start into the analysis. I have no numbers for gameday 1 to compare them to, as the preseason files are full of crappy strategies. So I’ll just post the win probabilities for every matchup and the final outcome. I will just post the win probability of the home team, as it’s not that hard to get the one from the away team, and it saves some numbers. I also don’t print the points scored, as this would be too much I assume. Beginning with week 2, I post the final win probability and the difference from the base probability for the home team.
Week 1
Yellowknife @ Baltimore 23-24, 38.1%
Philadelphia @ Colorado 17-19, 75.4%
Sarasota @ Chicago 20-9, 41.9%
Orange County @ Arizona 33-23, 42.3%
New Orleans @ San Jose 21-30, 37.7%
Honolulu @ Austin 17-37, 81.6%
Week 2
Colorado @ Baltimore 33-27, 44.7%, +1.8%
Sarasota @ Yellowknife 6-17, 80.2%, -1.2%
Chicago @ Philadelphia 19-17, 85.2, +0%
Honolulu @ Orange County 6-41, 82.0%, -1.8%
Arizona @ San Jose 24-13, 35.1%, +2.0%
Austin @ New Orleans 16-45, 61.5%, +11.0%
Week 3
Chicago @ Yellowknife 0-54, 90.2%, -2.4%
Baltimore @ Liberty 17-51, 66.3%, -0.6%
Colorado @ Sarasota 24-29, 48.9%, -3.0%
New Orleans @ Arizona 20-17, 60.1%, +5.6%
Orange County @ Austin 31-24, 64.9%, +11.6%
San Jose @ Honolulu 7-35, 75.8%, +2.8%
Week 4
Yellowknife @ Colorado 24-16, 62.3%, -2.0%
Baltimore @ Chicago 16-3, 37.9%, -7.4%
Philadelphia @ Sarasota 24-23, 60.7%, -0.2%
San Jose @ Orange County 20-17, 91.0%, +0.4%
Honolulu @ New Orleans 21-24, 83.0%, +5.2%
Arizona @ Austin 29-33, 74.1%, +3.6%
Week 5
Sarasota @ Baltimore 7-40, 60.9%, +3.6%
Philadelphia @ Yellowknife 23-24, 73.5%, +2.4%
Colorado @ Chicago 20-3, 30.7%, +2.4%
New Orleans @ Orange County 7-27, 69.7%, -4.8%
Austin @ San Jose 35-12, 26.7%, -6.2%
Arizona @ Honolulu 36-12, 54.3%, -0.4%
Week 6
Orange County @ Yellowknife 33-30, 54.9%, -4.8%
San Jose @ Colorado 10-37, 83.2%, -4.8%
New Orleans @ Philadelphia, 21-24, 51.7%, -1.0%
Chicago @ Austin 16-30, 87.4%, -2.8%
Baltimore @ Arizona 14-21, 66.5%, -0.8%
Honolulu @ Sarasota 17-23, 69.5%, +0.2%
Week 7
Orange County @ Baltimore 13-33, 44.7%, +5.0%
Honolulu @ Chicago 35-9, 41.3%, -5.2%
Sarasota @ Arizona 7-31, 63.1%, -5.6%
Yellowknife @ San Jose 34-27, 33.1%, +0.0%
Colorado @ New Orleans 24-26, 63.1%, -0.4%
Philadelphia @ Austin 17-7, 76.4%, +2.6%
Week 8
San Jose @ Baltimore 16-6, 68.1%, +1.8%
New Orleans @ Yellowknife 23-17, 68.3%, +3.8%
Austin @ Colorado 42-16, 64.5%, +0.6%
Arizona @ Chicago 17-22, 43.7%, +4.6%
Sarasota @ Orange County 20-24, 68.5%, -8.4%
Philadelphia @ Honolulu 12-21, 61.9%, -5.2%
Week 9
Arizona @ Philadelphia 34-13, 57.7%, -1.0%
San Jose @ Sarasota 14-20, 80.4%, -1.2%
Chicago @ Orange County 13-34, 89.4%, +0.8%
Baltimore @ New Orleans 13-25, 79.4%, +1.0%
Yellowknife @ Austin 23-17, 61.1%, +4.8%
Colorado @ Honolulu 3-6, 55.3%, -1.2%
Week 10
Austin @ Baltimore 20-24, 46.5%, -2.2%
Honolulu @ Yellowknife 14-40, 77.6%, +1.8%
Arizona @ Colorado 16-27, 66.7%, -6.4%
Orange County @ Philadelphia 13-24, 47.3%, +1.6%
Chicago @ San Jose 6-16, 69.5%, +0.2%
Sarasota @ New Orleans 24-30, 62.1%, -4.2%
Week 11
New Orleans @ Chicago 6-34, 41.7%, +4.2%
Austin @ Sarasota 13-14, 61.3%, +5.0%
Yellowknife @ Arizona 13-46, 52.3%, -0.2%
Colorado @ Orange County 15-9, 76.0%, +5.4%
Philadelphia @ San Jose 27-9, 47.3%, -10.8%
Baltimore @ Honolulu 14-10, 67.5%, +1.4%
So this are the final scores, the win probabilities for the games and the difference the home team had in win probability compared to the base win probability. There were some big changes in testing for some games, some are pretty close or even the same probability as the base was. Differences < 2% may also be just the fluctuation of the sim, but it’s hard to say. As already stated, I just used the numbers I got as given, and calculate with them the final sim testing rankings.
The best sim testers in the league…
full stop
are the guys of...
SARASOTA SAILFISH
Congrats.
Here is the final ranking
The Sailfish finish just close before Baltimore who also do an amazing job. Sarasota is quite average in its home games, but are taking their job super serious at away games it seems. Baltimore is the most balanced teams, they always seem to find a few percentage to boost their chances. Austin does amazing at home games, but should overthink what they are doing for away games. Philadelphia is similar to Sarasota, but only a bit less successful for both home and away games. New Orleans is on the same train as Austin, but can’t find the enormous success Austin is finding for their home games. Chicago is the last team who improves their win probabilities from the baseline. Honolulu, Yellowknife and Arizona all are pretty close at even, just a little below the 0. The three teams that are left, are all pretty bad in improving their win probability (maybe my sim runs did all just not go into their favor). San Jose at least is improving in away games, but struggles at home. Colorado isn’t really doing well in either, that’s why they are that low. The most surprising thing in that ranking for me was the last position. I didn’t expect the Orange County Otters being last in this ranking. I honestly expected them to battle for the #1 spot in this rankings. But they had some games were their win probability sunk far below the baseline. That sums up and find them only in the last spot here.
tl&dr
Sarasota, maybe: First, we don't have an entirely sim testing department. Second, by the time we play you our "WR who switched to QB" will be max passing stats and be as useful as young QB Cue. Third, when you actually test win percentage in the sim its a super low correlation between changing strategies and effectiveness in winning. So uh, I'm fine with not having a sim testing department. Fourth, I admire you attempting to bash a team's strategy for "doing sim testing" but if you're going to bash a team's strategy, it should be any team but Sarasota. I'm the best sim tester in the league, full stop. If we're running something its because its the optimal strategy for our team. But I mean, go off.