They look at every single play made (and not made) on video. Every one. They watch the video and plot the plays right there on the computer screen. They then compare each players ability to make plays. I’ve explained this before, but let’s say with nobody on base there’s a hard ground ball hit six feet to the left of second base. Let’s say 20% of all shortstops turn that play into outs. Well, if you make the play, you get a +.80 (because you’re making a play that 80% of shortstops do not make). If you don’t make that play you get a -.20 (you’re NOT making a play that 20% of shortstop make). That, in a nutshell, is the plus/minus system. Add up the plusses and minuses, and you should get a pretty precise number of plays that the fielder makes against average. Whatever subjectivity they have in their intensive and painstaking and methodical approach to studying defense seems to me to be NOTHING compared to the subjectivity that you and I get watching baseball even night after night after night. They look at every single play. More than look, they CHART every single play. More than chart, they CONCENTRATE on the defense for every single play. You and I don’t do that. Scouts don’t do that. Announcers, sportswriters, managers, general managers, nobody else does this. We may watch every single play (probably not), but we’re watching those plays in the larger context. We’re watching the pitcher, the hitter, the base runner, the umpires, the fans, the game. They are not. They are watching only defense. It’s a different thing. Much more at the link