+- International Simulation Football League (https://forums.sim-football.com)
+-- Forum: League Office (https://forums.sim-football.com/forumdisplay.php?fid=7)
+--- Forum: Sim Info (https://forums.sim-football.com/forumdisplay.php?fid=476)
+--- Thread: State of the Sim Transfer: Feb. 10th. (/showthread.php?tid=29881)
State of the Sim Transfer: Feb. 10th. - JPach - 02-10-2021
Hi everyone!
Now that we're almost halfway through the first season of the new sim era, we wanted to put out a sort of 'state of the sim transfer' newsletter.
We understand that there's not been much communication since the initial rollout and to that end, we would like to open up a semi-regular correspondence with you all. For this first newsletter, we thought we'd run through how we approached redesigning the sim, and what our priorities are for now.
Firstly, we want to describe the testing process that we used to design and balance archetypes, game planning, and other such aspects of the sim. Much like GMs now, we are definitely limited in the extent that we can't learn the exact impact of certain things on win rate. As you may have heard, examining the effect of pretty much anything on win rate requires a crazy amount more work than it did before. As an example of that, you would need to sim 67 seasons to be able to detect a jump in one team’s win rate from 50 to 55% with 90% confidence. In order to detect a 2% increase in win rate, you'd need to sim about 400 seasons worth of games to be 90% confident!
While we did not invest the time to sim everything over a huge number of seasons, we were able to very reliably test things that one would assume will lead to more wins. Higher YPC, YPA, completion rate etc., are all metrics where the sample size is dependent on the number of plays in a season, not the number of games played. These serve as good approximations for what helps a team win, and it is substantially easier to detect consistent changes in these metrics with a sample size of only a few seasons.
Using these kinds of metrics, our main aim was to create a league environment that was more in line with what a typical NFL season produces. We also paid close attention to the stats that each position was getting. We wanted to be close to real-world professional football in all aspects, meaning total scoring, passing/rushing totals, and stat distribution among positions.
However, there are a few issues that we had to deal with when taking this realism-driven testing approach, which I'll go into detail here.
1) Team totals vs player totals - Our rosters are a lot more concentrated than NFL rosters. The most extreme example of this is in the passing game. In the 2019 NFL season, there was 1 player with a >30% target share on his team. In the ISFL, a league with less than half the number of teams, we see more like 8 per season. So that leaves us with a dilemma. Do we want realistic numbers for QBs (and therefore realistic passing totals overall) or realistic numbers for individual WRs? Because we can't have both! And this actually applies to all positions. Do we want realistic total passing/rushing/tackle/sack/interception numbers, or realistic individual player numbers, despite the fact that our players play by far a greater % of snaps than the average NFL players do?
We decided that the priority should be with the overall numbers. If we tried to keep overall passing numbers suppressed in order to make sure no WR ever gets close to 2k yards for example, that would open up a whole load of issues elsewhere. So the priority was simulating a game as close as possible, accepting that individual records are likely to be higher than those in the NFL. While NFL QB records are likely to closely resemble those in the ISFL, we are expecting receiving records to be outpaced quite quickly.
2) How to test a system that doesn’t exist yet? When testing pretty much everything, we had to guess what archetypes users would select, what strategies/depth charts teams would use and what people would allocate their TPE into. For each of the 35 archetypes, we came up with around 40 builds for each, reflecting different levels of TPE and different build paths that one might take. That left us with a build repository of 1,442 different theoretical players. We then took the current list of ISFL and DSFL players and used their TPE to pick one of the theoretical players to adopt before testing. And it turns out we assumed a much more even spread of archetypes than ended up being the case!
3) How to get around issues with the way the sim works that are out of our control? The programming of DDSPF21 definitely has some ‘quirks’ that mean the default performance we were seeing was not desirable. Whether that was with DEs being far too powerful just by virtue of being listed as a DE or running QBs regularly recording negative yardage seasons. For the former, we spent a lot of time playing around with weight for DE/DT/OL in an attempt to make them more balanced. For the latter, we had to identify individual plays or even actions that were causing the issue, and tweak them until we came up with running QBs that were at least more effective than a running back.
All in all, we logged around 800 combined hours in DDSPF21, as well as probably 800 more in spreadsheets and debates, coming up with the basic rules and building blocks for the new-era ISFL.
And now that we can see how our 300 or so users have used and continue to use those building blocks, we are continuing to test with the idea of coming up with hotfixes, tweaks to playbooks/strategy options, and backend balancing patches. On that last point, we would love to avoid having to make messy archetype changes, but we are continuing to monitor balance to see if any will be necessary in the offseason.
Given the tradeoffs that we had to make in order to generate gameplay that resembles the NFL, we are generally happy with how the season has played out so far. Many of the games have come down to the wire and feature a good balance of exciting big plays (which the former sim had an unrealistically low number of) and defensive stops. However, there are definitely a few areas that we are unhappy with, sharing in some of the frustrations that many of you have voiced. In particular, we are looking into addressing the following:
1) Rushing. The running game is far too boom or bust, and we hold our hands up completely that we focused far too much on balancing the overall end of season numbers and not enough time looking at individual play by plays.
While end-of-season rushing averages will look fairly realistic, this is being made up of far too many long runs and micro/negative yardage plays. There seems to be very little in the way of consistent medium-yardage runs in the 4-10 yard range or so. We are in the early stages of working out a fix for this and are getting promising results that would not require any changes to players’ builds, but will dramatically reduce the frequency of both the micro-yardage runs and the long runs, and will result in more realistic rushing overall. This will hopefully mean that next season will see RBs perform more evenly week to week and can be relied on more.
2) Penalties - The way these are programmed is not ideal, and is almost completely out of our control to fix. Penalties seem to be auto-accepted in all situations, can result in automatic safeties and can result in a less favorable situation than if the penalty was declined. The most egregious of these is DPI, which apparently only gets called on completed passes, and was resulting in the ball getting pulled back to the line of scrimmage for a first down. We have implemented a fix for DPIs specifically in time for Week 6’s ISFL games by editing the ‘Penalties.xml’ file. DPI will still get called, but the next play will start from wherever the previous play finished up rather than erasing the previous play. I can only apologize to those affected negatively by this up until now.
Additionally, penalty numbers in the DSFL appear much higher than in previous seasons, leading to games that feel dominated by penalty yardage and more frequent occurrences of weird penalty glitches as described above. We are looking into potential solutions to this by modifying player personalities which would also require no changes to players’ builds.
We have really enjoyed helping shape this new era of the ISFL sim and we are pretty honoured to have been given the opportunity. We’re also under no illusions that our work is done, and we look forward to implementing fixes over the offseason that continue to improve the quality of gameplay. I sincerely believe that we will be able to iron out all the kinks and end up with a product that makes this league the best in the business.
We are more than happy to answer any and all questions and are 100% open to constructive feedback and suggestions about things we might have missed or ideas to help improve this game that we love playing even more. So fire away!
RE: State of the Sim Transfer: Feb. 10th. - DatSmolBoi - 02-10-2021
Yeah this is too long to read where is the tdlr
RE: State of the Sim Transfer: Feb. 10th. - siddhus - 02-10-2021
:sjock:
RE: State of the Sim Transfer: Feb. 10th. - Kotasa - 02-10-2021
Content
RE: State of the Sim Transfer: Feb. 10th. - Faded - 02-10-2021
neat
RE: State of the Sim Transfer: Feb. 10th. - Bayley - 02-10-2021
@JPach We have implemented a fix for DPIs specifically in time for Week 6’s ISFL games by editing the ‘Penalties.xml’ file. DPI will still get called, but the next play will start from wherever the previous play finished up rather than erasing the previous play. I can only apologize to those affected negatively by this up until now.
Does this mean a DPI would be declined in all instances?
RE: State of the Sim Transfer: Feb. 10th. - GuitarMaster116 - 02-10-2021
Thank you sim team for all the hard work and effort you have put into this, and I think myself and many others really appreciate getting an update regarding the state of the sim and hearing some occurring issues are being worked on!
RE: State of the Sim Transfer: Feb. 10th. - Swanty - 02-10-2021
(02-10-2021, 03:41 PM)Bayley Wrote: @JPach We have implemented a fix for DPIs specifically in time for Week 6’s ISFL games by editing the ‘Penalties.xml’ file. DPI will still get called, but the next play will start from wherever the previous play finished up rather than erasing the previous play. I can only apologize to those affected negatively by this up until now.
Does this mean a DPI would be declined in all instances?
Annoyingly, we can't stop the sim from accepting all the penalties, but the way it was working before, it was counting it as a 0 yard penalty from the line of scrimmage. So we turned on the 'Tack On' parameter for that penalty, so that the 0 yards applies from the end of the play instead of the start. None of the penalties seem to work from 'the spot of the foul'. And this penalty only seems to get called on complete passes as far as we can tell.
So yeah, it'll be effectively like the penalty on the completed pass was declined, except the next play will be a first down (if it wasn't already)
RE: State of the Sim Transfer: Feb. 10th. - UberBJ - 02-10-2021
Glad you posted this, the transparency and methods being used is good to know from an outside perspective.
RE: State of the Sim Transfer: Feb. 10th. - Oles - 02-10-2021
(02-10-2021, 03:33 PM)DatSmolBoi Wrote: Yeah this is too long to read where is the tdlr
TLDR is they spent 800 hours, made a shit load of spreadsheets, and still didn't test enough to actually fix issues but instead focused on making sure WRs wouldn't get 2k yards. Meanwhile we see egregious issues like DPI, penalties and all that are just now being fixed even though the sim team spent a lot of time in it.