Real question to the group…and I know there will be several different perspectives/answers on this:
What do you think is the “right” or “perfect” mix of advanced metrics/stats, and good old-fashioned coaching/intuition/eye test? Meaning, what percentage of each of the two categories (metrics vs intuition) would you allocate to your own formula that would yield the best results in your opinion??
And I know that there could be other categories, but for this exercise I opted for simplicity with just the two choices.
I think it necessarily depends on the result you are trying to predict. If you want to produce a 30 game season and have the highest % accuracy, you probably want almost entirely stats. But basketball is a very high variance game, so you will be optimizing to solve a different problem if you want to - say - predict the winner of one single elimination tournament game. Now, if you can make me a predictor that is robust to small sample sizes and high variances, then you can likely make some $$$ - but you would likely already be in a finance or stats phd program rather than posting on here.
I am in finance. But not a PHD candidate that’s for sure!!
It’s a generic question - not a great one - just trying to see where people fall on the stats vs. eye test argument. I think I go 75% eye test, 25% metrics.
And I meant no disrespect re the PhD comment. Rather, the idea of optimizing prediction models and specifying your loss function etc etc are big questions that have serious rewards if we can answer!
That’s a great question. I’m not a numbers, person never have been in my professional or my sporting/coaching year. Mainly because I believe you can find or create a stat to tell whatever story you want to hear. I also changed my mind after hearing Billy Beane speak once and he admitted that money ball at it’s purest does not work once you reach a closed series format, ie. the playoffs.
So that’s a long intro to say I’m probably 65/35 Eye test to metrics.
The metrics that people use are typically characterized by an omitted variable bias.
Consider all of the variables that go into a seemingly simple statistic like field goal percentage. A fourth option on offense who scores primarily on offensive rebounds and drive-and-dish opportunities is not easily compared to the first option on offense who is frequently asked to “make a play” at the end of the shot clock when running the offense didn’t yield a quality shot option earlier in the possession.
I think somewhere in this range is where I fall. I’m old and old school enough to dislike what advance metrics have done to all sports, but still young and innovative enough to understand and appreciate their value. Metrics are a tool, not the be all end all. Both your eyes and statistics can lie to you. I still trust my eyes more, so 75% 25% makes sense to me.
On top of this, there are also data selection issues. For example, in March what matters is predicting the outliers. However, when fitting a model - or otherwise choosing a statistic - you likely would choose one that does not heavily weight outliers (or even ignores the outlier data). And you do this because you don’t want to be predicting upsets in every single game during the regular season; because the sample size of a regular season is so much larger, you expect that the distribution of the sample of results will approach the true distribution.
I think I would start as 75/25 and then would gradually be seduced by the metrics argument and the larger data set, and end up closer to 60/40 by the end of the season.
That’s why I landed in the 65/35 range. Because I have found random metrics, that I like especially as a coach to help prove a point of emphasis I want my team to learn. But when it comes down to the tough decisions I’m going with eye/gut test.
Data helps, but I think it should only be supplemental to the eye test. A lot of things data can’t account for. And even if you have data, there are multiple assumptions or take aways you can get from that same set of data which is where the eye test can come in
Probably an obvious point here (my specialty), but for me, there’s a feedback loop. Does the eye test backup the metrics? Do the metrics back up the eye test? Where do they agree? Where are they discordant?
I was thinking this year - and did not have the time to do it- of making a March madness bracket based on high variance stats. In other words, looking at teams who have large outlier statistics that are heavily variable game to game. For example, really low pace teams, teams that shoot really high or really low % of 3pt shots. Maybe next year…
YOU HAVE TO DO THIS!! So foreign to me but so cool. Can you do it for the Sweet 16 or is it to much of a time investment? I have no clue…but would be cool to see and discuss.
I will see if time comes up, but if anyone is interested I would probably want to look at:
Pace (lower possessions, lower sample size)
%of shots team takes that are 3pt shots (really high or really low outliers)
variance/standard deviation in 3pt shooting
free throw rate (this is a weird, dynamic stat because it depends on how your other shooting is and how the reffing is
Not sure if there are other interesting ones to include.
The really robust way to handle this would be to actually look at the big stats and then the game to game variance of the remaining teams, and see which ones have the highest variance. Alas, dayjobs etc
For a person who knows what they’re doing and seeing, I think player evaluations would be almost all eye test. How a guy shoots, his form, his quickness, how he defends, how he reads situations, his passing, his vision, stuff like that. You mix in 10% stats to eliminate subconscious biases. The Daryl Morey story about when he scouted Jeremy Lin, the stats and physical testing out of college put him in an elite athlete category, but he didn’t pass the scouts’ eye test. He (and others) surmised that it was because Lin is Asian and scouts just didn’t have anyone to compare him to and they didn’t believe their eyes that an Asian dude was capable of doing what he was doing.
Now, if you’re talking basketball strategy, I would rely more on stats and metrics. If your guy shoots 50% on catch and shoot corner 3s but 30% on catch and shoot from anywhere else and 20% on 3s off the dribble, you design plays to get him catch and shoot corner 3s or you have your defense really cover him in the corner but they can sag off to help if he’s anywhere else. If you have Armaan Franklin hitting 25% from 3 but dominating off the dribble in the mid range, do more of that and less of the 3.
Then you go back to the eye test to try to figure out why he’s succeeding in one area and not another.
If he was acing the physical testing stuff and not passing the “eye test” then that just sounds like old school bias, and not the kind of bias that’s implicit in the eye test.
I’ll do this at the macro level, team analytics. I’m gonna operate off imperfect memory here. In prior NCAA tourneys, I seem to recall the selection committee selecting teams with strong NET, but not great records. There were critiques of these selections. But when you tracked how those teams did, they validated their selection.
This year’s edition of this dynamic? Michigan. Rough win-loss record, but solid NET. Their selection over other teams was criticized. And now look at Michigan: Sweet 16.