*Running QBs:To Check The Box or Not Check The Box

*Running QBs:To Check The Box or Not Check The Box - Printable Version

+- International Simulation Football League (https://forums.sim-football.com)
+-- Forum: Community (https://forums.sim-football.com/forumdisplay.php?fid=5)
+--- Forum: Media (https://forums.sim-football.com/forumdisplay.php?fid=37)
+---- Forum: Graded Articles (https://forums.sim-football.com/forumdisplay.php?fid=38)
+---- Thread: *Running QBs:To Check The Box or Not Check The Box (/showthread.php?tid=21239)

Pages: 1 2 3

*Running QBs:To Check The Box or Not Check The Box - iStegosauruz - 04-16-2020

[div align=\\\"center\\\"]Background and Methodology[/div]
@timeconsumer simulated 14,400 games so that you wouldn’t have to. Recently a post has gone around on the forums talking about rules that need to be changed. In that, someone brought up the rule that states:

Quote:VII.A.4 All mobile QBs should be checked as running QB within the sim.

That got us to thinking: what is the impact to checking that box? Timeconsumer dug into that question, running the numbers to try to find the answer to that question. In doing so, however, he ended up digging pretty deeply into Orange County Otters quarterback Franklin Armstrong’s contributions to their run game.

He ran 36 different sets of simulations, each containing 400 games. Of those 400 games, 200 were at home and 200 were on the road. The matchup was constant every time: Orange County Otters v. New Orleans Second Line. For consistency, he used the week 5 sim file from the S21 NSFL season. In the actual season the Otters played the Second Line that week, with New Orleans winning on the road 22 to 17. This week was chosen because it was a week where both teams 4-0 on the season and were coming into the game with their best strategies to counter each other.

The 36 sets of simulations were broken down further into 3 different sets of 12. The 3 overarching sets of tests focused on different playbook variations. The first set used the Balanced/Spread combination that the Otters came into the game running on offense. The second set used the same percentages for the strategy, but swapped the formation to being Spread all of the time. The third set used the same percentages for the strategy, but swapped the formation to being Balanced all of the time.

Within each of those 3 sets of 12 tests are 6 different tiers of speed values that were tested. Essentially, he tested how Armstrong performed under each of the strategy variations at 75 speed, 79 speed, 80 speed, 85 speed, 89 speed, and 90 speed. Each of those tiers were tested with the “running quarterback” box within the sim checked and then run again with the box not checked. It is important to note that Armstrong currently has a speed rating of 79. His quarterback archetype is “Game Manager,” which maxes its speed at 80.

From those sims he pulled the Otters’s win percentage, Armstrong’s rushing attempts per game, Armstrong’s rush yards per game, Armstrong’s yards per carry per game, and Armstrong’s passer rating per game.

[div align=\\\"center\\\"]Data
[Image: Mb0WuVd.png]

Balanced/Spread Strategy

[Image: cIavcYD.png]

[/div]
Using the balanced/spread strategy they employed during their week 5 game against the Second Line and with Armstrong’s speed at his current level of 79, the Otters won the matchup 44.5% of the time when the “running quarterback” box was not selected in the sim. During those games Armstrong rushed an average of 6.2 times per game for an average total of 28.9 yards and 4.7 yards per carry. His passer rating for that batch of simulations was 85.8. At the same level of speed - 79 - when the box is checked in the sim the Otters won the matchup with the Second Line 37.5% of the time. During those games Armstrong rushed an average of 8.8 times per game for an average yards per game of 20.7, an average yards per carry of 2.4, and an average quarterback rating of 83.7. Utilizing the same strategy and with Armstrong having the same speed level he did during that point of the season, there is a 7% difference in the Otter’s win percentage in the matchup with the Second Line based on whether the “running quarterback” box is selected or not. When the box isn’t selected, Armstrong rushed 2.6 times less per game, however, he rushed for 8.2 more yards per game on average with a 2.3 better yards per carry average and a 2.1 difference in quarterback rating.

When Armstrong’s speed is bumped up to 80 - which is the cap for his archetype of quarterback - the Otter’s win percentage and Armstrong’s rushing performance change drastically. When he has 80 speed and the “running quarterback” box is not selected, the Otters won the matchup 13.5% less times than they did when the only change to the conditions was that Armstrong’s speed was at 79. The 1 speed increase also caused an average of a 28.93 yard per game difference in Armstrong’s total rushing yards and an average yards per carry difference of 4.7. His quarterback rating dropped from 85.8 to 81.2, a 4.6 difference.

When you look past the 80 speed benchmark, the running quarterback becomes more effective. At 85 speed with the “running quarterback” box checked, the Otters won an equivalent amount of games to the control grouping - 79 speed with the box not checked. Armstrong’s rushing performance changed slightly. His rushing attempts per game increased by 2.7, his rush yards per game increased by 0.7, but his yards per carry dipped by 1.4 - from 4.7 to 3.3. His rating also declined by 1.5 points, from 85.8 in the control grouping to 84.3 in the 85 speed/box checked grouping.

The Otters win percentage only climbs as his speed increases when the “running quarterback” box is checked, however. At 89 speed the Otters won the matchup 1.5% more times than they did in the control or the 85 speed grouping. Armstrong’s rating increased from 85.8 in the control and 84.3 in the 85 speed grouping to 88.7. This is a 2.9 point and 3.4 point change in rating respectively. At 90 speed the Otters won the matchup 51.3% of the time - a 6.8% difference from the control group.

What all of this data shows is that under the strategy the Otters employ - the Balanced/Spread style - the optimal speed rating for Armstrong is 79 and the “running quarterback” box should not be checked. This is counterintuitive to the belief all attributes should be capped, however, because there is a drastic difference in performance between 79 and 80 speed when the “running quarterback” box is not checked. After 80 speed - a mark that Armstrong cannot get to under his current archetype - the optimal strategy for the Otters is to continue to raise speed but to make sure the “running quarterback” box is checked.

[div align=\\\"center\\\"] [Image: XGWoO3z.png]

[/div]
[div align=\\\"center\\\"]Spread

[Image: nFgB0tz.png]

[/div]
Conceptually in football we identify the “spread” as a style offensive that thrives under a mobile quarterback. This is not entirely true for the sim, however. In the spread, the Otters experience a very similar phenomenon to the one they experience in the balanced/spread offensive. For any of the speeds that Armstrong can achieve with his archetype that were tested (75, 79, or 80), the optimal strategy is to not check the “running quarterback” box in the sim and to have him stop putting TPE into speed when it hits 79. When that strategy is employed the Otters won their matchup with the Second Line 41.3% of the time. The only time that the Otter’s win percentage in the spread increases past this value is when Armstrong has either 89 or 90 speed and the “running quarterback” box in the sim is selected.

Armstrong’s quarterback rating in the spread follows a similar trend. When his speed is less than or equal to 80, he puts up the best quarterback rating at 79 speed with the “running quarterback box” not selected. When he surpasses 80 speed his quarterback rating continually rises as his speed rises, but only when the “running quarterback” box is selected.

[div align=\\\"center\\\"] [Image: HDaX70V.png]

[/div]

[div align=\\\"center\\\"]Balanced

[Image: DAkLkNH.png]

[/div]

Under a purely balanced system, the trend with the Otters win percentage and Armstrong’s quarterback rating are similar, but not entirely equivalent, to that seen in the balanced/spread system and spread system. Using just a balanced system the optimal strategy for the Otters using speed values that Armstrong can achieve under his archetype is to have him stop at 75 speed and select the “running quarterback” box in the sim. Under that strategy, the Otters won their matchup with the Second Line 41.3% of the time. This is marginally better than having him stop at 79 speed and leaving the “running quarterback” box unchecked - a 0.5% difference.

At speed values past what Armstrong can achieve - so anything over 80 - the optimal strategy is to have him max his speed attribute and to select the “running quarterback” box in the sim. It is not a completely consistent trend however.

Armstrong’s quarterback rating continually improved as his speed value improved and was higher in situations where the “running quarterback” box was selected. This does break the trend somewhat of his quarterback rating being higher when the box was not selected and his speed was less than or equal to 80.

[div align=\\\"center\\\"] [Image: xV3XSaU.png]

[/div]
[div align=\\\"center\\\"]Conclusions [/div]
1. There is a clear difference between 79 and 80 speed in terms of a running quarterback vs a non-running quarterback box selection in the sim.
2. From 80 speed and up the running quarterback is more effective under most circumstances, however this might not be completely due to running.
3. The running box does appear to somehow add in extra runs that are not related to the playbook. We are unsure about what run type is being called in these situations.
4. It is confusing that the passer rating is increasing as quarterback speed increases with the running quarterback box selected in the sim. This cannot be due to the sack percentage because those are not factored into quarterback rating. It seems to have an effect on how the quarterback does in the metrics that do factor into that rating. In theory, we think it may be due to completion percentage - more speed meaning more of an ability to escape pressure and let the play develop - but we didn’t track that so its just a theory.
5. There is a serious question about whether the league needs to find a solution to the 79 speed/game manager archetype loophole Armstrong has been employing for the last few seasons. The Otters are running a close to ideal strategy for this combination with the balanced/spread strategy and experience a 7% difference in win percentage by using that strategy and not selecting the “running quarterback” box.
6. This subject requires further exploration, this is just the first part of what should be a topic of conversation around the league.
7. What it looks like is that the sim was built for quarterbacks to be good throwers until 79 speed, runners at 80 speed and up, and good runners once they hit around 85 speed.
[div align=\\\"center\\\"]Notes[/div]
1. This was/is by no means a hit piece on any particular player in the league. Armstrong was just the best control quarterback to use to test the theories.
2. Timeconsumer has been nice enough to allow me to have 100% of the media grading for this. That being said, I do think he should receive some credit for this by the graders. I did the write-up and charts while he did the simulations.

*Running QBs:To Check The Box or Not Check The Box - ADwyer87 - 04-16-2020

im very dumb. So the conclusion is....the box helps if you're above 80 speed?

*Running QBs:To Check The Box or Not Check The Box - speculadora - 04-16-2020

this is very cool. I never actually bothered testing the win rates when we made this call. Moonlight came to me and said he wanted to run, I tested some builds, 80 speed sucks. very interesting to see it backed up in data.

that being said, I'm not sure that I'd necessarily describe it as a loophole. looking at the table, if we're using the same strategy at 75 speed, we only drop just 8 extra games in a 200/200 sample size. it takes about 300 sims to normalize win rate so you could debate the difference there is mostly noise, especially given how much variance I've seen running the same strat twice in a row

*Running QBs:To Check The Box or Not Check The Box - timeconsumer - 04-16-2020

(04-16-2020, 09:38 PM)ADwyer87 Wrote:im very dumb. So the conclusion is....the box helps if you're above 80 speed?

Yeah kinda but not necessarily because you run better. We think you pass better also but I didn't capture that data well enough. I just added in passer rating to look for an outlier and well, we might have found it.

*Running QBs:To Check The Box or Not Check The Box - timeconsumer - 04-16-2020

(04-16-2020, 09:39 PM)speculadora Wrote:this is very cool. I never actually bothered testing the win rates when we made this call. Moonlight came to me and said he wanted to run, I tested some builds, 80 speed sucks. very interesting to see it backed up in data.

that being said, I'm not sure that I'd necessarily describe it as a loophole. looking at the table, if we're using the same strategy at 75 speed, we only drop just 8 extra games in a 200/200 sample size. it takes about 300 sims to normalize win rate so you could debate the difference there is mostly noise, especially given how much variance I've seen running the same strat twice in a row

Loophole might not be the best word. It's just really interesting the difference between 75/79/80 and it's worth exploring further to see if there is a significant difference between a max pocket passer at 75 speed and a max game manager at 79 speed or if they are roughly equal. If equal I'd say change the archetype max to 79 to ensure anybody who creates as a game manager is setup for success. The whole point behind the archetype is to make it a decent rusher and decent passer and 80 speed ain't that (sans box), is 79 ideal or is it too much? IDK yet.

*Running QBs:To Check The Box or Not Check The Box - Opera_Phantom - 04-16-2020

:eyes:

*Running QBs:To Check The Box or Not Check The Box - Faded - 04-16-2020

(04-16-2020, 06:30 PM)iStegosauruz Wrote:[div align=\\\"center\\\"]Background and Methodology[/div]
I simulated 14,400 games so that you wouldn’t have to.

Lol nice

*Running QBs:To Check The Box or Not Check The Box - PMoney - 04-16-2020

Only 14,400 games? Is this even a real analysis?

But seriously, really interesting to see and in a very impressive time scale. I only posted it to clear up wording and these are really interesting results.

*Running QBs:To Check The Box or Not Check The Box - timeconsumer - 04-16-2020

(04-16-2020, 09:56 PM)PMoney Wrote:Only 14,400 games? Is this even a real analysis?

But seriously, really interesting to see and in a very impressive time scale. I only posted it to clear up wording and these are really interesting results.

Autoclickers are gamechangers

*Running QBs:To Check The Box or Not Check The Box - PMoney - 04-16-2020

(04-17-2020, 02:12 AM)timeconsumer Wrote:Autoclickers are gamechangers

iSteg has spoiled us with his simming