*4th Downs and Surrender Punts

[infinitempg]

I know this should probably go into Statistical Analysis but I've spent way too much time on this for it to be buried there. Sue me.

[div align=\\\"center\\\"][/div]

[div align=\\\"center\\\"]An Analysis on Fourth Down Decisions and Surrender Punts[/div]

I think everyone who's paid attention to the live streams of our sims has noticed a strange but common trend:
[div align=\\\"center\\\"][/div]
The sim loves 30(ish) yard punts. The sim's affinity for this has been a cause of my own personal suffering, as FIVE 30-40 yard punts cost Micycle McCormick the season punting average record (you can see that game here). Simply changing the 4 shortest punts (32, 32, 36, and 37) to field goal attempts would have allowed McCormick to achieve the glory of overtaking Turk's 50.9 yards per punt record with 51.0 yards per punt. (And maybe I'd get in the Hall of Fame...)

But putting my personal problems aside, we do not see this trend in the modern NFL or CFB. Kickers have become a lot stronger in recent years, and so teams are more willing to try the long kick. Of the 15 seasons with the most attempted 50+ yard field goals, only 2 come from before 2007:
[div align=\\\"center\\\"][/div]
(For reference, a 50 yard field goal is kicked from about the 32 or 33 yard line.)

NFL kickers are also much more accurate from distance nowadays, exceeding 50% accuracy (via FiveThirtyEight):
[div align=\\\"center\\\"][/div]

Okay, this all makes sense. But our sim was released in 2016, so why doesn't it seem to reflect the decisions of real football teams of the time? It's time to do some analysis.

[div align=\\\"center\\\"]Data Collection[/div]

So obviously, all of my data comes from the uploaded indexes. I scraped all the play by play tables as well as the boxscore tables, combined them together, and created a giant database of every single play in NSFL history. There were some weird bugs (specifically a weird one with Arizona) that I haven't been able to explain, but I was able to fix them away later I didn't scrape the DSFL because (a) there's a fairly clear skill discrepancy (think #CollegeKickers) and (b) it would be extra work—and I’m lazy. Also, this includes pre-season games, despite the fact that most pre-season games don't include set strategies and depth charts. All of my data and code is available on my GitHub, by the way (link at the bottom).

At this point, I have to thank timeconsumer and WhiteCornerback for managing to find the post-Ultimus sim files for S5, S7, S8, and S13, which were all corrupted on the indexes. With the permission from 37thchamber, I was able to recompile and re-upload them.

[div align=\\\"center\\\"]So what do teams do?[/div]

With this database of all plays, I split off all the possible endings to a drive (touchdown, field goal, turnover on downs, interception, fumble, and safety) to see their frequency:
[div align=\\\"center\\\"][/div]

Nearly half of all drives end in a punt! However, this seems to match fairly well to NFL data (taken from Jon Bois' Surrender Index video, you'll see this a lot later), where about 43% of all drives end with punts.

[div align=\\\"center\\\"][/div]

The rest of the stats also seem to follow nicely. Point for DDSPF, I guess.

[div align=\\\"center\\\"]Field Position Matters[/div]

Obviously, the decision a team makes on 4th down depends on where they are in the field, and how far they have to go to get a new set of downs. The three choices a team can make are to go for it, kick a field goal, or punt. The longest field goal attempted in the NSFL is 65 yards, which by the way, was made. This was kicked at the 48 yard line! We can't expect too many attempts from this far, however.

So what does DDSPF decide to do?

[div align=\\\"center\\\"][/div]

There are some pretty clear demarcations in this plot. If the team is anywhere behind the opponents 40, the sim will almost certainly punt the ball away. Between the opponent's 31 and 40, the sim attempts a field goal about 10% of the time, still mostly choosing to punt. This is, for reference, anywhere between a 48 and 57 yard field goal - of which we expect most of this range to be either tries or field goals. Inside the opponent's 29, the sim is almost certain to kick the field goal, just as expected. The strange spot is at the opponent's 30, where the sim still decides to punt the ball nearly 20% of the time instead of kicking the 47 yard field goal!

Here's the NFL, for comparison (again from Bois' video):

[div align=\\\"center\\\"][/div]

This distribution is much smoother than our simulated counterpart. The percentage of tries appears to stay consistent throughout the field, while the sim has discontinuities at the 30 and 40. We see almost no punts from within the 32, a significant change. Also of interest is the complete lack of tries inside the opponent's 3 yard line like we see in the NFL.

[div align=\\\"center\\\"]The Dead Zone[/div]

We'll take the sim's clear line in the sand and call it the "Dead Zone". How does the distance to the first down marker affect the sim's decision making?

[div align=\\\"center\\\"][/div]

As we move further and further from the marker, the percentage of punts reasonably increases. Frustratingly, the percentage of field goals remains remarkably consistent with distance:

[div align=\\\"center\\\"][/div]

It seems like 4th and less than 5 is the sweet spot to at least attempt a conversion, but the sim simply refuses to acknowledge the merits of long field goals.

You may have noticed that I cut this graph off at 25 yards. This is because there is such a small sample of plays with 4th and more than 25 also in the Dead Zone:

[div align=\\\"center\\\"][/div]

What this data tells me is the sim was programmed with clearly marked boundaries and rules, as opposed to the more gradual shift we see in the real world. There's a "phase change" (for lack of a better word) at the 30 yard line, and another at the 40, with small variations depending on the specific distance to reach the first down. Is this the most accurate way to program the sim? Meh. Is it easy? Probably easier than defining the probabilities on logistic curves like the real NFL appears to behave.

[div align=\\\"center\\\"]Go Fer It![/div]

So, what does affect the sim's decision to try a conversion? We did see for the Dead Zone that distance played an effect, but does this pay off?

[div align=\\\"center\\\"][/div]

Yes. There's a pretty consistent ~60% success rate on 4th and 5 or less, which linearly decreases after until we get to the dreaded small sample size:

[div align=\\\"center\\\"][/div]

Looking at the line graph, we can see the relationship between success rate and distance appears to be linear and inverse:

[div align=\\\"center\\\"][/div]

I have no qualms with this. It makes complete sense to model a 4th down attempt this way. As an aside - there are two successful conversions of 4th and 20+: once by NOLA down 31-20 with 1:04 left and once by COL down 48-24 with 7:14 left. Why even attempt there???

Let's also take a quick look at success rates by field position, as this will be important later:

[div align=\\\"center\\\"][/div]

We can see here that the average success rate closer to the opponent's endzone is about 50%, though failures tend to pile up the further pinned back the offense is. I can't really think of a good reason why this happens outside of such attempts being made out of desperation.

[div align=\\\"center\\\"]Take the Points![/div]

The next logical place to look is at field goals. Surely looking at the accuracy numbers will tell us why long field goals aren't attempted:

[div align=\\\"center\\\"][/div]

Heck, I don't see anything weird here. From 50-55 yards, NSFL kickers average better than their NFL counterparts (80% to 65%), and are even comparable at over 60 (though again small sample size must be accounted for):

[div align=\\\"center\\\"][/div]

So to quickly summarize where we are - going for it has a strong inverse dependence on distance and sim league kickers are actually way more accurate from 50+ than they should be. This doesn't seem to add up logically, though as we all know the sim doesn't exactly excel in that field.

[div align=\\\"center\\\"]Punts[/div]

Well I guess we should look at punts then. We're mostly concerned about the Dead Zone, but let's see how field position affects if a punt is returned, fair caught, or left for a touchback:

[div align=\\\"center\\\"][/div]

This follows what we expect, but it forces us to once again question the logic of the sim. First of all, there are clearly no attempts to "coffin corner" the ball when punting close to the endzone, as evidenced by the ridiculous amount of touchbacks within the opposing 40. We also see that while punt returners are braver the further away they are from their own endzone, there are still a decent amount of fair catches. As we can see here, the percentage of fair catches (or no returns) track well with the percentage that are actually returned:

[div align=\\\"center\\\"][/div]

The other obvious question here is why even bother punting inside the 40?! Nearly 100% of all punts inside this range are touchbacks, resulting in a net field position change of 10-20 yards. Also remember that the sim provides a high amount of field goal accuracy from this same range (> 60%). I just don't follow the logic here.

Let's just do a quick back-of-the-napkin calculation here. In the Dead Zone, the sim punts about 50% of the time, goes for it about 40% of the time, and kicks the field goal 10% of the time. We can assume a 50% success rate for tries and a 60% rate of successful field goals. We can also take a very rough estimate of 3 points per successful try based on the pie chart from earlier (42% being touchdowns and 28% being field goals). Plugging this all in:

[div align=\\\"center\\\"][/div]
[div align=\\\"center\\\"][/div]

On average, not punting results in 1.56 points per drive. If we then extend this to the drives where the sim un-reasonably punted (about 1244 times), we get 1940 points punted away. There have been 320 total regular season and preseason games played (14 seasons of 18, 4 seasons of 17), so each game we have a total of 6.06 points punted away per game!

It's hard to say whether this certainly affected the outcomes of specific seasons and playoff runs, but we can definitely bet that the decision to punt inside the opponents 40 has cost teams games. If only there was a term for willingly giving up points like this...

[div align=\\\"center\\\"]Surrender Punts[/div]

A surrender punt is defined as a punt that signifies the kicking team's complacency with losing. These tend to happen in the opponent's side of the field, in a situation where a team needs points but instead of going for it on 4th down they decide to punt the ball away for little net field position gain. As a fan, surrender punts can be annoying - but as a player they are down right enraging. Everyone on that field knows that the offense should be aggressive, after all the worst consequence of it is losing 20 or so yards of field position, but the one person in charge decides to play it conservative and your offense never sees the ball again.

In order to track the sadness of a surrender punt, Jon Bois created a Surrender Index (again, you can watch his video here). The key factors here are the score differential, the time remaining in the game, the field position of the team punting, and how far the team would have to go for a first down. So if your team was up by 20 with 2 minutes to go on a 4th and 15, your surrender index would be nearly 0. But if you had a 4th and 2, down by 3, at your opponent's 37? Your surrender index is going to rise astronomically. I'm going to skip over a lot of the details, but the biggest impact on the Surrender Index is the time remaining, which goes up cubically as a factor of the time since half time.

Once I've gotten all the details out from the index, it becomes fairly simple to calculate the surrender index for all punts in the history of the NSFL. Here is a graph comparing the distribution of surrender indexes in the NSFL versus the NFL (which I got here from Andrew Shackelford):

[div align=\\\"center\\\"][/div]

These match pretty well! Make sure you note that the x-axis here is in a logarithmic scale. We see here that the NSFL has a lot of very un-sad punts compared the the NFL - the lowest surrender index in the two leagues are 0.03 and 0.2 respectively. I'm not really sure what causes that but it's interesting to note. The rest of the distribution match pretty evenly however.

If we look at the high end of the surrender punts, we find that there are actually 6 punts in NSFL history with a higher surrender index than the highest NFL surrender index of 659.7. We'll go over each from least sad to most sad.

[div align=\\\"center\\\"]6) S7W8 - COL @ NOLA
Surrender Index: 685.83
Boxscore[/div]

Time: Q4, 4:04 left
Score: 3 - 38
Punting Team:
Down and Distance: 4th and 1
Field Position: 34 yard line
Result: Punt by McCormick, M. of 34 yards. Touchback.
Final Score: 3 - 38

I don't know if this should be seen as a surrender punt or a "let us get on the buses" punt, but either way punting on a 4th and 1 is sad. Glad to know that my player was involved in one of the saddest punts of all time.

[div align=\\\"center\\\"]5) S7W7 - YKW @ SJS
Surrender Index: 695.33
Boxscore[/div]

Time: Q4, 8:30 left
Score: 23 - 20
Punting Team:
Down and Distance: 4th and 3
Field Position: 32 yard line
Result: Punt by Blewitt, I. of 32 yards. Touchback.
Final Score: 26 - 20

To be fair to the Sabercats, 8:30 is a lot of time. Unfortunately for them, the decision to punt instead of going for it on 4th and 3 at the 32 yard line allowed the Wraiths to drive down the field, burn off 4 minutes of game clock, and kick a field goal to extend their lead to 6. Had San Jose just attempted the 49 yard field goal this result could have been a whole lot different.

Also funny that two of the saddest punts ever took place a week after each other.

[div align=\\\"center\\\"]4) S3W10 - PHI @ YKW
Surrender Index: 708.05
Boxscore[/div]

Time: Q4, 3:16 left
Score: 14 - 17
Punting Team:
Down and Distance: 4th and 5
Field Position: 33 yard line
Result: Punt by Harris, S. of 33 yards. Touchback.
Final Score: 14 - 17

Once again a situation where the sim undervalues field goals - Philly could have easily tied this game up with a 50 yard field goal, but noooo. They only net 13 yards and never see the ball again. There's certainly no guarantee that the Wraiths can drive down the field with 3 minutes to go (because the sim makes Andy Reid look like a competent time manager), but punting it away completely costs them this game.

[div align=\\\"center\\\"]3) S1W1- BAL @ COL
Surrender Index: 730.99
Boxscore[/div]

Time: Q4, 2:03 left
Score: 20 - 23
Punting Team:
Down and Distance: 4th and 3
Field Position: 35 yard line
Result: Punt by Harris, S. of 35 yards. Touchback.
Final Score: 20 - 23

The very first week of football had the third saddest punt of all time - talk about making a first impression. The Yeti had just scored a touchdown 3 minutes prior and made the bold choice to go for 2, managing to convert it for a 3 point lead. I'm not sure if the sim makes decisions based on the point differential being 1, 2, or 3, but that would be cool to look into...... okay let's look into it

[div align=\\\"center\\\"][/div]

It appears that if tied, the sim is content punting it away in the Dead Zone. Down by 1, the sim gets fairly aggressive and goes for it. Down by 2 the sim becomes mysteriously shy and punts it away nearly 80% of the time, despite a field goal having the same effect. Down 3, the sim reverts back to a less unreasonable 30% for each option before completely giving up on FGs from 4-6 points down. The sim is also most aggressive down 9? Either way, we can see that going for 2 instead of kicking the field goal was statistically the worse option for the Yeti despite all common sense and real football analytics saying to go for that 2. Sim gonna sim I guess.

[div align=\\\"center\\\"]2) S11W7- YKW @ PHI
Surrender Index: 739.11
Boxscore[/div]

Time: Q4, 4:24 left
Score: 21 - 17
Punting Team:
Down and Distance: 4th and 1
Field Position: 35 yard line
Result: Punt by ® Fulture, K. of 35 yards. Touchback.
Final Score: 21 - 24

This is the first time a surrender punt paid off for a team! A truly atrocious decision to not go for it on 4th and 1 resulted in a quick three and out for the Liberty defense. On the ensuing punt, Andre Bly Jr. put on his hero cape and returned Forfeit's punt 68 yards to the house to take a 3 point lead. The Liberty defense managed to stand pat the rest of the game and allow such conservative play calling to be rewarded.

[div align=\\\"center\\\"]1) S6W4- YKW @ NOLA
Surrender Index: 930.99
Boxscore[/div]

Time: Overtime, 12:29 left
Score: 27 - 27
Punting Team:
Down and Distance: 4th and 2
Field Position: 32 yard line
Result: Punt by Lombardi, J. of 32 yards. Touchback.
Final Score: 27 - 27

This punt is 200 points sadder than the second saddest punt of all time, but yet it kind of pays off? It's really strange and the Second Line should have been punished if not for a heroic interception by the Second Line LB Luke Washington in his own endzone on the ensuing drive.

The punt itself is already truly atrocious - 4th and 2 at the 32. There's a 60% chance you succeed on a try, and an 80% chance that the 49 yard field goal goes through the uprights. Also did I mention it's OVERTIME?! The time multiplier plays a huge role in this calculation, and rightfully so. Any points win this for the Second Line, but they were content to punt and managed the survive with a tie by the skin of their teeth.

Even more amazingly, this tie allowed the Wraiths to sneak into the playoffs above the Hawks, as otherwise they would have been tied at 7-7 with Baltimore having the better points for (is that still the second tiebreaker? If it's conference record then I guess it doesn't actually matter then).

***

If you've made it this far, thanks for reading! I think it's abundantly clear that the sim is cowardly when it comes to the opponent's 30s, and it's fairly well reflected in the higher surrender indexes. There are some truly horrifying ones here at the top, but interestingly the top two don't punish the aggressor that hard. Not sure what kind of karma that is.

TL;DR: Sim is a coward and it's programmed a bit too simply, perhaps?

GitHub repo: https://github.com/infinitempg/surrender_punt

All the files, scraped data, and images are located here. Feel free to poke around at my spaghetti code!

[manicmav36]

Nerd.

But for real, this is awesome. Great work!

[Raven]

I love this

[timeconsumer]

Can do all this but can't figure out how to make a powerpoint work on OBS.....you're a rare breed Mr. Infinite

[37thchamber]

GitHub repo: https://github.com/infinitempg/surrender_punt

All the files, scraped data, and images are located here.

Also, someone give this man a bonus. This data is going to be very useful beyond this article.

[Jay_Doctor]

This makes sense. It feels like so many punts have been taken that make you scratch your head

[infinitempg]

Can do all this but can't figure out how to make a powerpoint work on OBS.....you're a rare breed Mr. Infinite

obs is dum dont @ me

[ckroyal92]

[div align=\\\"center\\\"]5) S7W7 - YKW @ SJS
Surrender Index: 695.33
Boxscore[/div]

Time: Q4, 8:30 left
Score: 23 - 20
Punting Team:
Down and Distance: 4th and 3
Field Position: 32 yard line
Result: Punt by Blewitt, I. of 32 yards. Touchback.
Final Score: 26 - 20

To be fair to the Sabercats, 8:30 is a lot of time. Unfortunately for them, the decision to punt instead of going for it on 4th and 3 at the 32 yard line allowed the Wraiths to drive down the field, burn off 4 minutes of game clock, and kick a field goal to extend their lead to 6. Had San Jose just attempted the 49 yard field goal this result could have been a whole lot different.

Also funny that two of the saddest punts ever took place a week after each other.

Surrendered so badly we just quit the game?

[infinitempg]

Surrendered so badly we just quit the game?

I KNEW THERE WAS A MISTAKE SOMEWHERE

but also basically yeah

Also I just noticed that 4 of the top 6 surrender punts involve the team playing the Wraiths... what voodoo magic do you have up there @Bigred1580

[Bigred1580]

I KNEW THERE WAS A MISTAKE SOMEWHERE

but also basically yeah

Also I just noticed that 4 of the top 6 surrender punts involve the team playing the Wraiths... what voodoo magic do you have up there @Bigred1580

Cooter Magic