Using a Dense Neural Network to Predict Wins Based on QB Stats
Hey there, its wonderful_art and I’m presenting today another data science project for all those nerds like me in the ISFL who love this type of stuff. The focus of this article (and series most likely moving forward) is deploying a machine learning model, specifically a perceptron dense neural network, onto ISFL stats to gleam further insights on how those stats reflect reality – and what they say about the player overall.
This particular project is the start of a longer series on predicting wins. I’m hoping to build larger more complex models to follow in this initial footstep, but I’m being agile and iterating through MVPs as I go. So, this is just the beginning, in other words, of some fun stuff I’m hoping to do with stats and the league resources.
Methodology
The goal is to generate QB stats reliably from web-scraping indexes, normalize the statistics and feed them into a dense neural network in the hope of building a sensitive model to accurately predict wins based solely off the QB statistics. Through the use of a Google Colab notebook, I’ve been successful, building a model with 0.8852 root mean squared error – essentially only off by 0.9 wins when predicting on average. Pretty good, I’d say – although there’s certainly room for improvement. You can check out the Google Colab notebook here.
What’s included? Essentially the model looks at every quarterback’s stats since S5 through S24, specifically at completions, attempts, yards, average completion pct, longest throw, touchdowns, and interceptions. Each one is normalized for that season’s passing stats, meaning each season is unique for their given field – and it allows us to predict a quarterback’s final season wins before the end of the season (a model based on actual numbers, not normalized, wouldn’t have this ability).
What does the dense neural network look like? It takes in a series of features (all the stats listed above) and feeds them through 3 dense perceptron layers – one with 64 hidden layers, the second with 32 hidden layers, and the final layer is just a signal firing layer that produces a prediction for each value in the data – hopefully the correct wins for that quarterback in that season. Each epoch, the model learns what weights to assign its nodes and connections to provide the correct final output from the last layer. In total, there are 2,625 parameters the model can adjust to get the correct final number – the correct number of wins for that quarterback’s stats.
Here’s a plot of the model:
It runs for 100 epochs, continually optimizing based on a loss function called mse – mean squared error. It takes the sum of squared differences between our target variable and the predicted value – the target being the wins that season for a QB, and the predicted value the result of the model’s prediction. This loss function works well to emphasize drastic errors, which due to being squared are the focus of optimization. It makes continual adjustments to these parameters to optimize this mean squared error. For communicating the error, its much more valuable to take the root – which then gives us a number at scale to the values to understand what error we are seeing. In this case, roughly 0.79 wins account for error. It initially started about 7 wins, and over 100 epochs for each of the 19 seasons (~1900 total epochs) it reduced to 0.79 error on wins predicted vs actual.
Here’s the model summary:
S25 Quarterback Predicted End of Season Wins
Now that I’ve outlined the model and its training, here’s the results for the current S25 group. Don’t forget wins are predicted off current stats for the entirety of the season.
How does this compare to the eye test? Well there’s some interesting predictions here based solely off of QB stats – awarding Jack the highest predicted wins out of any QB, by a fairly decent margin too, is not well explained by the data available at first glance. What follows is my explanation for the model, but I should not this is effectively a black box – the model relies on those ~2,500 parameters finely tuned to this task to predict results, and so my explanation may be flawed.
So while award Jack 11 predicted wins? Not the highest yards, attempts, completions, pct, or touchdowns. The key is that model highly values quarterbacks who have a strong all around game - and it doesn't care how often they throw. Jack's put up the second highest touchdowns, second lowest interceptions, and second best completion percentage.
But then why is Pheonix so low and not second (or first for that matter) with a league leading 10 touchdowns? While the model seems to value touchdowns (obviously), it really loves quarterbacks who are efficient in the passing game and have a high completion percentage. So having a sub 55% completion rate seems to affect the model's predicted output based on league history.
Why is Fujiwara not higher? Despite the most yards, the model seems to disadvantage quarterbacks who are heavily used and relied on for an air raid style of play. Fewer attempts at bat most likely means fewer opportunities for a devasting pick. The model predicts that quarterbacks who are used not too much and not too little, and who perform well when they’re called with efficient play and not too many interceptions, will provide their team more wins than a stud QB who’s always called on to dish out the ball downfield. Between heavy usage and high interceptions, Fujiwara is not ranked highly by the model's predicted wins.
Let's look into some data analytics and see how Predicted Wins correlates to the major statistics for each quarterback. Note that Passer Rating was not included in the model, but it has a high correlation with the output - meaning the sim likely uses a linear regression model that resembles this dense neural network final model for deciding passer rating. And that QBR is a decent metric for evaluating quarterbacks via the index.
S24 Quarterback Predicted Wins vs Actual Wins
We can also check how my explanations line up with last season’s S24 quarterback results:
This result appears to line up with my explanation – McDummy had few interceptions, high efficiency, and not a crazy number of attempts. Whereas the model viewed Banks as used far too much with too much risk inherent in that type of play – high yards, high TDs, but also high interceptions. The model explains that while it worked for Banks last season, across its 14 seasons of passing data, more often than not, that style costs wins.
Expected Wins vs Actual Wins (WinContribution) for Historic ISFL Quarterbacks
Using the model, however, we can also investigate historic data on who contributed the most and least to their team's victories - at least from the model's perspective. This is essentially the equivalent to asking who was the most underrated quarterback in the league - who otherwise put up good stats to the model but didn't receive the wins that would be expected and predicted given those statistics.
Here's the results:
There are two stats above: the WinContribution_Sum derived from the sum of all season's differential between the model's predicted wins and the actual wins for that quarterback; and the WinContribution_Mean which is the average for that quarterback's career. Note that its just pure addition behind the WinContribution metric, meaning that players with longer careers and many seasons of over-performing compared to team win totals would quickly gather higher numbers over the course of their careers.
So who's the most underrated according to the model? There's a number of candidates but it settled with Austin Copperhead's Easton Cole with just over 22 predicted wins over his career actual win total (by the model's estimate he should have roughly 69 career wins versus his actual 47). This was especially emphasized in S16 as a rookie when the Copperheads won 0 games, but the model predicted 5, and in S18 when the Copperheads won a respectable 5 games, but Cole's play was valued at 11 wins. Despite the team around him catching up to Cole's play, he's consistently performed at a rate higher than his total wins would reflect.
What about the most underrated season? Here are the outliers there:
Top 5 Highest and Lowest WinContribution QB seasons:
You'll note that the lowest 5 WinContribution QBs are all from successful seasons on teams, when the team had success on the field. The model generally under values these quarterbacks, likely predicting closer to a mean predicted wins around 9 which likely increases its odds of success.
Team Analysis
And finally, we can look at how each team's quarterbacks have faired via a swarm plot:
Clearly the model doesn’t have a full picture and a better model to predict wins would need to capture more than just the quarterback’s passing stats. Future projects on this space I’m planning includes incorporating more player statistics in a hope to provide a more fulsome team picture, and potentially run fun hypotheticals on team changes. Otherwise I’d like get more timeseries data and include a more predictive distribution based on player modelling and TPE – the latter isn’t even included in this model and likely would play a key role in any predictive analytics for the league. Stay tuned for more updates and work on this angle as I continue diving into it.
Hey there, its wonderful_art and I’m presenting today another data science project for all those nerds like me in the ISFL who love this type of stuff. The focus of this article (and series most likely moving forward) is deploying a machine learning model, specifically a perceptron dense neural network, onto ISFL stats to gleam further insights on how those stats reflect reality – and what they say about the player overall.
This particular project is the start of a longer series on predicting wins. I’m hoping to build larger more complex models to follow in this initial footstep, but I’m being agile and iterating through MVPs as I go. So, this is just the beginning, in other words, of some fun stuff I’m hoping to do with stats and the league resources.
Methodology
The goal is to generate QB stats reliably from web-scraping indexes, normalize the statistics and feed them into a dense neural network in the hope of building a sensitive model to accurately predict wins based solely off the QB statistics. Through the use of a Google Colab notebook, I’ve been successful, building a model with 0.8852 root mean squared error – essentially only off by 0.9 wins when predicting on average. Pretty good, I’d say – although there’s certainly room for improvement. You can check out the Google Colab notebook here.
What’s included? Essentially the model looks at every quarterback’s stats since S5 through S24, specifically at completions, attempts, yards, average completion pct, longest throw, touchdowns, and interceptions. Each one is normalized for that season’s passing stats, meaning each season is unique for their given field – and it allows us to predict a quarterback’s final season wins before the end of the season (a model based on actual numbers, not normalized, wouldn’t have this ability).
What does the dense neural network look like? It takes in a series of features (all the stats listed above) and feeds them through 3 dense perceptron layers – one with 64 hidden layers, the second with 32 hidden layers, and the final layer is just a signal firing layer that produces a prediction for each value in the data – hopefully the correct wins for that quarterback in that season. Each epoch, the model learns what weights to assign its nodes and connections to provide the correct final output from the last layer. In total, there are 2,625 parameters the model can adjust to get the correct final number – the correct number of wins for that quarterback’s stats.
Here’s a plot of the model:
It runs for 100 epochs, continually optimizing based on a loss function called mse – mean squared error. It takes the sum of squared differences between our target variable and the predicted value – the target being the wins that season for a QB, and the predicted value the result of the model’s prediction. This loss function works well to emphasize drastic errors, which due to being squared are the focus of optimization. It makes continual adjustments to these parameters to optimize this mean squared error. For communicating the error, its much more valuable to take the root – which then gives us a number at scale to the values to understand what error we are seeing. In this case, roughly 0.79 wins account for error. It initially started about 7 wins, and over 100 epochs for each of the 19 seasons (~1900 total epochs) it reduced to 0.79 error on wins predicted vs actual.
Here’s the model summary:
Code:
Model: "sequential_56"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_168 (Dense) (None, 64) 512
_________________________________________________________________
dense_169 (Dense) (None, 32) 2080
_________________________________________________________________
dense_170 (Dense) (None, 1) 33
=================================================================
Total params: 2,625
Trainable params: 2,625
Non-trainable params: 0
_________________________________________________________________
S25 Quarterback Predicted End of Season Wins
Now that I’ve outlined the model and its training, here’s the results for the current S25 group. Don’t forget wins are predicted off current stats for the entirety of the season.
Code:
Predicted Wins Team Name
---------------- ------ --------------------
11.291 SJS Jack, M.
10.783 NO Slothlisberger, B.
10.136 COL McDummy (C), W.
9.193 HON Skywalker (C), L.
9.07 OCO Ramza (R), S.
8.659 AUS Cole (C), E.
8.406 BER Kaepercolin, N.
8.029 NY Howitzer (R) (C), S.
7.609 YKW Jack, C.
7.452 CHI O'Donnell, G.
6.5 AZ Cue (C), J.
5.794 PHI Phoenix (C), B.
5.459 BAL Fujiwara (C), C.
3.506 SAR Banks II, D.
How does this compare to the eye test? Well there’s some interesting predictions here based solely off of QB stats – awarding Jack the highest predicted wins out of any QB, by a fairly decent margin too, is not well explained by the data available at first glance. What follows is my explanation for the model, but I should not this is effectively a black box – the model relies on those ~2,500 parameters finely tuned to this task to predict results, and so my explanation may be flawed.
So while award Jack 11 predicted wins? Not the highest yards, attempts, completions, pct, or touchdowns. The key is that model highly values quarterbacks who have a strong all around game - and it doesn't care how often they throw. Jack's put up the second highest touchdowns, second lowest interceptions, and second best completion percentage.
But then why is Pheonix so low and not second (or first for that matter) with a league leading 10 touchdowns? While the model seems to value touchdowns (obviously), it really loves quarterbacks who are efficient in the passing game and have a high completion percentage. So having a sub 55% completion rate seems to affect the model's predicted output based on league history.
Why is Fujiwara not higher? Despite the most yards, the model seems to disadvantage quarterbacks who are heavily used and relied on for an air raid style of play. Fewer attempts at bat most likely means fewer opportunities for a devasting pick. The model predicts that quarterbacks who are used not too much and not too little, and who perform well when they’re called with efficient play and not too many interceptions, will provide their team more wins than a stud QB who’s always called on to dish out the ball downfield. Between heavy usage and high interceptions, Fujiwara is not ranked highly by the model's predicted wins.
Let's look into some data analytics and see how Predicted Wins correlates to the major statistics for each quarterback. Note that Passer Rating was not included in the model, but it has a high correlation with the output - meaning the sim likely uses a linear regression model that resembles this dense neural network final model for deciding passer rating. And that QBR is a decent metric for evaluating quarterbacks via the index.
S24 Quarterback Predicted Wins vs Actual Wins
We can also check how my explanations line up with last season’s S24 quarterback results:
Code:
Predicted Wins Team Actual Wins Name
---------------- ------ ------------- ----------------------
13.024 COL 14 McDummy, W.
9.115 OCO 9 Armstrong, F.
8.995 AUS 7 Cole, E.
8.683 SJS 9 Jack, M.
8.459 NO 9 Slothlisberger (R), B.
8.34 HON 8 Skywalker (R), L.
8.266 CHI 7 O'Donnell, G.
8.144 SAR 12 Banks II, D.
8.048 YKW 6 Jack (R), C.
6.018 AZ 9 Cue, J.
5.946 BAL 3 Fujiwara, C.
2.865 PHI 3 Phoenix, B.
This result appears to line up with my explanation – McDummy had few interceptions, high efficiency, and not a crazy number of attempts. Whereas the model viewed Banks as used far too much with too much risk inherent in that type of play – high yards, high TDs, but also high interceptions. The model explains that while it worked for Banks last season, across its 14 seasons of passing data, more often than not, that style costs wins.
Expected Wins vs Actual Wins (WinContribution) for Historic ISFL Quarterbacks
Using the model, however, we can also investigate historic data on who contributed the most and least to their team's victories - at least from the model's perspective. This is essentially the equivalent to asking who was the most underrated quarterback in the league - who otherwise put up good stats to the model but didn't receive the wins that would be expected and predicted given those statistics.
Here's the results:
Code:
UniqName WinContribution_Sum WinContribution_Mean
--------------- --------------------- ----------------------
Cole, E. 23.78 2.9725
Fitzpatrick, K. 20.3 4.06
Havran, C. 18.4 2.3
Applehort, R. 15.78 1.9725
McCormick, M. 15.57 3.114
Falconi, A. 15.5 1.72222
Cue, J. 13.24 3.31
Bronko, K. 12.34 3.085
Reese, A. 12.11 2.01833
Pennington, B. 11.95 2.9875
Noble, L. 11.68 3.89333
Jenkins, R. 11.43 1.63286
Pierno, N. 11.32 5.66
McDummy, W. 9.3 1.1625
Maximus III, B. 8.28 0.828
Christ, J. 7.96 1.13714
Blocksdale, A. 7.79 1.29833
Gambino, C. 7.14 1.785
Phoenix, B. 6.56 1.64
Francisco, S. 6.08 0.76
Bigsby, C. 5.42 0.602222
Fujiwara, C. 5.35 2.675
Wright, D. 4.52 0.904
Armstrong, F. 3.63 0.518571
LaFleur, F. 2.79 2.79
Oles, T. 2.19 0.73
O'Donnell, G. 1.66 1.66
Jack, ". 0.7 0.7
Banks II, D. 0.53 0.53
Rove, C. 0.43 0.43
Vega, Z. 0.37 0.37
Akselsen, M. -0.73 -0.121667
Orosz, C. -1.4 -0.466667
Boss, M. -3.63 -0.726
Showbiz, G. -3.73 -0.532857
There are two stats above: the WinContribution_Sum derived from the sum of all season's differential between the model's predicted wins and the actual wins for that quarterback; and the WinContribution_Mean which is the average for that quarterback's career. Note that its just pure addition behind the WinContribution metric, meaning that players with longer careers and many seasons of over-performing compared to team win totals would quickly gather higher numbers over the course of their careers.
So who's the most underrated according to the model? There's a number of candidates but it settled with Austin Copperhead's Easton Cole with just over 22 predicted wins over his career actual win total (by the model's estimate he should have roughly 69 career wins versus his actual 47). This was especially emphasized in S16 as a rookie when the Copperheads won 0 games, but the model predicted 5, and in S18 when the Copperheads won a respectable 5 games, but Cole's play was valued at 11 wins. Despite the team around him catching up to Cole's play, he's consistently performed at a rate higher than his total wins would reflect.
What about the most underrated season? Here are the outliers there:
Top 5 Highest and Lowest WinContribution QB seasons:
Code:
Name Team Season Actual Wins Predicted Wins WinContribution
----------- ------ -------- ------------- ---------------- -----------------
Noble, L. LV 5 1 8.49 7.49
Pierno, N. COL 6 0 6.65 6.65
Jenkins, R. CHI 20 4 10.25 6.25
Bronko, K. AZ 8 5 10.66 5.66
Cue, J. AZ 21 3 8.5 5.5
Code:
Name Team Season Actual Wins Predicted Wins WinContribution
--------------- ------ -------- ------------- ---------------- -----------------
Akselsen, M. YKW 7 13 8.69 -4.31
Christ, J. SJS 15 11 7.04 -3.96
Blocksdale, A. BAL 10 13 9.32 -3.68
Orosz, C. SJS 6 11 7.51 -3.49
Maximus III, B. NO 9 12 9.03 -2.97
You'll note that the lowest 5 WinContribution QBs are all from successful seasons on teams, when the team had success on the field. The model generally under values these quarterbacks, likely predicting closer to a mean predicted wins around 9 which likely increases its odds of success.
Team Analysis
And finally, we can look at how each team's quarterbacks have faired via a swarm plot:
-
What can we take away? For a team that has been historically very good, like the Otters, most of their quarterback seasons have fallen right around 0 WinContribution - most likely due to the fact that the model predicted them to have success - and they did. The WinContribution metric is better for identifying teams that have had a wide distribution of quarterback performance relative to their final standing - San Jose's QBs have ranged the entire distribution.
Clearly the model doesn’t have a full picture and a better model to predict wins would need to capture more than just the quarterback’s passing stats. Future projects on this space I’m planning includes incorporating more player statistics in a hope to provide a more fulsome team picture, and potentially run fun hypotheticals on team changes. Otherwise I’d like get more timeseries data and include a more predictive distribution based on player modelling and TPE – the latter isn’t even included in this model and likely would play a key role in any predictive analytics for the league. Stay tuned for more updates and work on this angle as I continue diving into it.
Quote:2093 words and research and data work