Catcher Framing Sample Size

PITCHf/x has made the effect that the strike zone has on the game more apparent. Catcher framing based on PITCHf/x results has helped fans better understand the effects that a good catcher can have on the game. Ian York attempts to find the catcher framing sample size needed to be considered reliable.

It is now well established that catcher framing the ability to catch pitches at the edges of the strike zone so that umpires are more likely to call them strikes is a very valuable skill, with some catchers contributing the equivalent of several wins per season from their framing skills alone. Several sites, such as StatCorner and Baseball Prospectus, list framing skills as part of their player evaluations, and framing has become a common part of discussions on baseball sites.

In one such discussion at recently, “Savin Hillbilly” raised the question of what sample size was adequate for assessing a catcher’s framing ability. After how many games is it realistic to judge a catcher? If a player has only caught 10 or 20 games, are the numbers remotely meaningful? How about 40, or 80?

To begin to answer the question, I measured framing with various sample sizes, to see how closely they match the value for the whole season. The short answer is that as you’d expect more is better, but 40 games or so should get you most of the way there.

I looked at the 2015 season for six catchers: Jonathan Lucroy, Yasmani Grandal, Wilson Ramos, Yan Gomes, Carlos Ruiz, and Salvador Perez. I chose them because they span the range from excellent (Grandal) to abysmal (Ruiz), they all caught at least 80 games in 2015, and they had reasonably similar framing numbers for 2014 as in 2015 (making it less likely that their framing ability changed significantly over the course of the 2015 season).

I calculated their full-season framing ability, which I express here as “extra strikes per 100 chances” (“ES/100”). (A “chance” is any called strike or ball.) I used “per 100 chances” rather than “per game”, because random sampling of chances across the season made the concept of a “game” fuzzy. Catchers generally have around 70 chances per game.

“Extra strikes” are calculated as described here. Briefly, I divide up the strike zone into sectors, ask what the league probability of a strike is in each sector, count the number of called pitches a specific catcher sees in each sector, and subtract the expected number of strikes (based on the league average) from the number he actually got.  

Here are the full-season framing numbers for our representative catchers:

Catcher Extra strikes/100 chances
Yasmani Grandal 2.60
Jonathan Lucroy 0.62
Wilson Ramos -0.17
Yan Gomes -0.41
Salvador Perez -0.82
Carlos Ruiz -1.61

Then, I randomly chose 500, 1,000, 2,500, or 5,000 chances (the equivalent of about 7, 14, 35, or 70 games), and calculated the ES/100. I did this 500 times for each sample size. As you would expect, the average of each group of 500 replicates was very close to the season average, but with fewer samples to look at the spread was wider:


In these boxplots, 50% of the estimates fall inside the boxes, and 90% fall between the whiskers. The red horizontal line across each chart shows the full-season value.

It is clear that having more samples is better. With just 500 chances, there is a wide spread of values; the 50% range spans about 1.5 ES/100 (meaning you have about a 50% chance of being within 0.75 ES/100 of the “real” value).

It isn’t much better with 100 chances (about 1 ES/100). By 2,500 chances, though, the 50% range is about 0.5 ES/100, and the 90% range is about 1 (so there is a 90% chance of coming within 0.5 ES/100 of the actual value). The numbers are better for 5,000 chances, but not hugely so.

Even the smaller samples have a decent chance of distinguishing broad tendencies. With 500 chances, Grandal barely overlaps with Ruiz even at the 90% range; there is an excellent chance that a very good framer could be distinguished from a terrible one with just a week’s worth of games. By 1,000 chances, Grandal is well separated from the league-average Gomes, and with 2,500 chances about a month’s worth of games the fairly good Lucroy could be reliably distinguished from the fairly bad Perez.

This is a very simplistic look at the question. Still, if we are looking for a rule of thumb, by the time a player has caught 40-50 games about half a season then his framing numbers are likely to be a reasonable estimate of his full-season averages.  

Follow Ian on Twitter @iayork.

All data compiled from PITCHf/x and