05-09-2024, 01:32 PM
(This post was last modified: 05-10-2024, 10:02 AM by wetwilleh. Edited 2 times in total.)
This is my first attempt at sim testing to dig out data, so I want to acknowledge at the very front that the data collection portion of it isn’t anywhere close to statistically significant. I say that mostly because I want to understand the process first before I get nuts with crazy complex scenarios and lots of sim running.
There was a conversation that spun up in the discord recently where there was talk about position swaps and position skill that got me digging in to find out how position skill was assigned.
It lead me to this balance thread: https://forums.sim-football.com/showthre...?tid=30480
If you look at that thread, it says that position skill ratings include a combination of position base, TPE, and XP value (where XP is the number of seasons played).
This had me wondering, what does Position Skill actually do inside the game? How much does it affect the Sim and do the changes you get when gaining TPE and XP change things significantly in your favor due to the Position Skill itself?
The Setup:
Using a DSFL sim file that completed Week 1, I updated every player to have 50 in every single attribute (strength, agility, endurance, etc…). The attempt here is to level the playing field to only look at positional skill. All rosters, strategies, etc.. are set the exact same way that they were set in Week 1 of the S47 DSFL season.
The Scenarios:
Having standardized Attributes (meaning, TPE is the same for everyone), I focused on just a single position, Quarterback, to run 4 different scenarios of set skill level and see what the stats seemed to tell.
I ran each scenario 50 times to get a reasonable amount of data.
Base Scenario - QB Skill level 1
Scenario 1 - QB Skill level 33
Scenario 2 - QB Skill level 66
Scenario 3 - QB Skill level 100
The Outcomes:
Gathering all of the player, team, and game data, the following is what the data has kicked out.
Game Data:
Across 50 runs, at 4 games per run, I collected the average scores. The X-axis was the game ID, so 4 games in a week, per run.
Above you see the home and away scores. As you can see, there isn’t a significant difference in scoring across the different scenarios. The average difference in scoring between the base scenario and the 100 Skill level scenario was between .5 and 6.5pts. Interestingly, Scenario 3 isn’t even the highest scoring average, so skill level doesn’t seem to directly correlate to more points.
Taking that same view into Team Wins, you end up with the same conclusion. Across the different scenarios, the difference in team wins at the different skillet levels was .005. Meaning, it wasn’t even close to creating a single win difference as you increase skill level.
Finally, I wanted to look at the QB stats themselves as the skill level was increased. Maybe the team wasn’t impacted overall, but how was the individual impacted?
I grabbed Plays, Pass Attempts, Pass Completions, Interceptions Thrown, Pass Yards, Pass TDs, and Pass Rating as my metrics per Quarterback.
When you look at all the different stats:
All in all, it does seem clear that individual performance is impacted by skill rating, but not at a rate that seems to be a significant increase. Instead, it’s smaller amounts with the exception of passing yards, but even the increase in passing yards didn’t translate into a clear win difference.
In Summation:
Using Position Skill rating, and connecting Exp to that, I believe that it has a pretty minor effect on the sim output. The numbers that I see above seem to indicate more that skill rating could be the tipping point of a close game rather than a metric that you would actually try and use for comparisons in a game winning strategy. If two players were playing in a tied game, you’d give a slight advantage to the more skilled one, but it would be so small that you couldn’t rely on that consistently given all the other factors in the sim.
So What Next?
As I mentioned at the very top, the intent of this work was more about figuring out the data I could get out of the sim and how to set my scenarios up. I was able to write some code that lets me set attributes and skills easily, which I think opens the door to deeper testing, as well as having the means to quickly put everything together (and know what I might want to see when I do).
In relation to position skill specifically, I think another deeper step would be to get more data. Both in the number of times you run, but also probably at an entire season level so that you can get averages across teams all playing each other multiple times.
Personally, I think I’m more interested in testing as it directly relates to TPE and Attribute gain, as that is going to have a much more significant impact. The fact that QB passing ratings were so low is certainly tied to the max value of 50 for all stats (on both the QB and the WR side).
Let me know your thoughts on the data above, but also I’m curious what different scenarios people would be interested in seeing more about.
There was a conversation that spun up in the discord recently where there was talk about position swaps and position skill that got me digging in to find out how position skill was assigned.
It lead me to this balance thread: https://forums.sim-football.com/showthre...?tid=30480
If you look at that thread, it says that position skill ratings include a combination of position base, TPE, and XP value (where XP is the number of seasons played).
This had me wondering, what does Position Skill actually do inside the game? How much does it affect the Sim and do the changes you get when gaining TPE and XP change things significantly in your favor due to the Position Skill itself?
The Setup:
Using a DSFL sim file that completed Week 1, I updated every player to have 50 in every single attribute (strength, agility, endurance, etc…). The attempt here is to level the playing field to only look at positional skill. All rosters, strategies, etc.. are set the exact same way that they were set in Week 1 of the S47 DSFL season.
The Scenarios:
Having standardized Attributes (meaning, TPE is the same for everyone), I focused on just a single position, Quarterback, to run 4 different scenarios of set skill level and see what the stats seemed to tell.
I ran each scenario 50 times to get a reasonable amount of data.
Base Scenario - QB Skill level 1
Scenario 1 - QB Skill level 33
Scenario 2 - QB Skill level 66
Scenario 3 - QB Skill level 100
The Outcomes:
Gathering all of the player, team, and game data, the following is what the data has kicked out.
Game Data:
Across 50 runs, at 4 games per run, I collected the average scores. The X-axis was the game ID, so 4 games in a week, per run.
Above you see the home and away scores. As you can see, there isn’t a significant difference in scoring across the different scenarios. The average difference in scoring between the base scenario and the 100 Skill level scenario was between .5 and 6.5pts. Interestingly, Scenario 3 isn’t even the highest scoring average, so skill level doesn’t seem to directly correlate to more points.
Taking that same view into Team Wins, you end up with the same conclusion. Across the different scenarios, the difference in team wins at the different skillet levels was .005. Meaning, it wasn’t even close to creating a single win difference as you increase skill level.
Finally, I wanted to look at the QB stats themselves as the skill level was increased. Maybe the team wasn’t impacted overall, but how was the individual impacted?
I grabbed Plays, Pass Attempts, Pass Completions, Interceptions Thrown, Pass Yards, Pass TDs, and Pass Rating as my metrics per Quarterback.
When you look at all the different stats:
- Play Count and Pass Attempts are clear indications that game strategy means much more here. The fact that the averages are all negative just means the game strategies impacted the plays that were even called.
- Pass Completions did increase, at an average of 4.76 a game.
- Pass Interceptions actually increased on average of .26, which is surprising because you would expect higher skill and more completions to be less interceptions. I can’t explain this other than to think that the sim has hidden logic tied to skill rating somehow.
- Passing yards increased by an average of 66.10 yards. This is a decent jump.
- Pass TDs increased by an average of .42. So a jump from 1 skill to 100 skill was still less than a 50% chance of throwing an additional TD in a game.
- Pass Rating increased on average by 7.8 points
All in all, it does seem clear that individual performance is impacted by skill rating, but not at a rate that seems to be a significant increase. Instead, it’s smaller amounts with the exception of passing yards, but even the increase in passing yards didn’t translate into a clear win difference.
In Summation:
Using Position Skill rating, and connecting Exp to that, I believe that it has a pretty minor effect on the sim output. The numbers that I see above seem to indicate more that skill rating could be the tipping point of a close game rather than a metric that you would actually try and use for comparisons in a game winning strategy. If two players were playing in a tied game, you’d give a slight advantage to the more skilled one, but it would be so small that you couldn’t rely on that consistently given all the other factors in the sim.
So What Next?
As I mentioned at the very top, the intent of this work was more about figuring out the data I could get out of the sim and how to set my scenarios up. I was able to write some code that lets me set attributes and skills easily, which I think opens the door to deeper testing, as well as having the means to quickly put everything together (and know what I might want to see when I do).
In relation to position skill specifically, I think another deeper step would be to get more data. Both in the number of times you run, but also probably at an entire season level so that you can get averages across teams all playing each other multiple times.
Personally, I think I’m more interested in testing as it directly relates to TPE and Attribute gain, as that is going to have a much more significant impact. The fact that QB passing ratings were so low is certainly tied to the max value of 50 for all stats (on both the QB and the WR side).
Let me know your thoughts on the data above, but also I’m curious what different scenarios people would be interested in seeing more about.