Screenshot at Jun 01 04-30-23

Tyranny of the Masses

Why developers should be weary of tracking player behavior.

“Big Brother is watching; not in an effort to control you but rather to learn from you. You’re not just playing videogames anymore. You’re actively providing feedback about what parts you like and which you don’t. How you play could ultimately help shape the future of videogame design.”

BioWare is just one of numerous development studios and publishers that have begun collecting anonymous player data. No identifying information is tied to the information harvested, so you don’t have to worry about things being traced back to you. You’re just a data point amongst millions.

-Erik Budvig

Were you worried that Bioware was not being influenced enough by the thousands of reviews, forum posts, emails, and tweets they receive for every single one of its games? Are you looking for a more impersonal way of communicating your gaming experiences with developers? Are you too lazy to write them an email with your comments and complaints?

If you are any one of these unfortunate people, then worry not my friends, for now you too can HELP SHAPE THE FUTURE OF GAME DESIGN. If, on the other hand, you’re one of those party poopers who still cares about antiquated 20th century concepts like ‘privacy,’ then you may still rest easy knowing that “you’re just a data point amongst millions.” In other words, Bioware’s data gathering efforts will somehow manage to give you unprecedented power while simultaneously making you an insignificant statistic. Makes perfect sense (statistically speaking, of course).

As you can probably tell, I’m more than a little baffled by the utopian declarations that have accompanied news of Bioware’s efforts to collect anonymous player data from Mass Effect 2. Gamers already influence the design decisions of mainstream developers  in a variety of ways, so it is simply absurd to imply that player tracking will somehow give “voice” to a previously disenfranchised demographic.

The question we should be asking is whether player tracking is good for Bioware and, consequently, good for people who want to continue playing ‘Bioware games.’ To be sure, I  absolutely get the appeal of collecting tracking data. Indeed, tracking a modest bit.ly link is pretty fun in itself (‘Ooooh look! I gots me 10 new clicks from Malaysia…clearly,  those guys know good Mario fan-art when they see it’), so I can’t even begin to imagine how great it must feel to be able to track one’s audience after years of working on a project as large and complex as ME2. But alas, the road to development hell is paved with good intentions, and I can’t help but worry that Bioware’s understandable desire to quantify player experiences might eventually backfire.

To understand the potential dangers of player tracking, we need to ask ourselves at least two questions: First, who gets to interpret the collected data? Second, how will this data influence the decision-making process of the interpreter?

The biggest brother

I don’t think it is too elitist to suggest that, as a general rule, artists do not produce their best work by worrying too much about the public’s (alleged) expectations. It was this overriding concern with fan service that made the completely arbitrary appearance of R2-D2 in The Phantom Menace seem like less than a terrible idea. This deference to “the audience” is also the reason that Nintendo decided to follow-up the Link of Wind Waker  with a more “mature,” conventional, and far less interesting version of Link in Twilight Princess.

So even if  Budvig’s claim that “Big Brother is watching” referred only to Bioware, that  would be reason enough to worry. But alas, Bioware is not the only one watching, and they will not have final say on how the data should be interpreted.  For Bioware depends on a bigger, badder brother; a passive-aggressive patron that has never made a secret of its desire to take your lunch money and leave you in tears. I’m talking about the biggest brother of all, which is to say Bioware’s publisher and parent company, EA.

If you’re reading this, chances are you already know a thing or two about EA’s history. When Trip Hawkins founded the company in 1982, his idea was to create a “different” kind of videogame publisher. His goals were incredibly idealistic for the time: EA would foster creativity, approach videogames as an “artform,” and treat game developers with the ‘respect’ that ‘artists’ deserve. Even the name “Electronic Arts” was a self-conscious attempt to convey these founding principles. As Hawkins explained in a 2007 Gammasutra article about the company:

“The original name had been Amazin’ Software. But I wanted to recognize software as an art form….So, in October of 1982 I called a meeting of our first twelve employees and our outside marketing agency and we brainstormed and decided to change it to Electronic Arts.”

But then Hawkins left the company in 1991, leaving a former Johnson & Johnson executive in charge. This is the point at which Electronic Arts began to transform itself into a “serious” publisher, abandoning its founding principles in the process.

Eric-Jon Rossel Waugh continues the story:

No sooner was Hawkins out the door than the acquisitions (and Madden milking) began.

[...]

The pattern to these acquisitions, if not universal, is infamous: find a company that made a really popular game, acquire the company and its properties; then set the team on churning out sequel after sequel to the game in question. Sometimes, likely not by design, the staff leaves or burns out, or one of the products sells poorly; the studio is closed or subsumed. Of EA’s acquisitions, only Maxis is known for retaining its autonomy and culture within the EA corporate structure, the jewel in EA’s crown.

EA seemed to have abandoned all of its founding principles and developed an attitude of rapid growth whatever the long-term cost, thereby setting a poor example for the rest of the industry.

And thus, the iconoclastic developer formerly known as Electronic Arts had completed its transformation into a faceless entity known simply as EA. Incidentally, EA was not the only multinational corporation to reduce it’s formerly descriptive name into an ambiguous acronym. According to Wikipedia, 1991 (the year of Hawkins’s departure) was also the year when Kentucky Fried Chicken changed it’s name to “KFC.” Conspiracy theorists claimed that the name-change came about because it’s genetically altered meat could no longer be considered “chicken.” Using that same conspiratorial logic, we may also say that Electronic Arts became “EA” once gamers realized that the company was no longer interested in promoting anything that could reasonably be called “art.”

Of course, my “theory” about EA’s name-change is a complete fabrication and, just like the KFC conspiracy theory, this will probably turn out to be false. But that’s entirely beside the point. The real problem for both of these companies is that they put themselves in a position that makes these rumors seem plausible to begin with. If KFC’s chicken still looked unmistakably like chicken, then no one would’ve developed a conspiracy theory about their use of KFC. Likewise, if EA hadn’t lost its way, perhaps there wouldn’t be a need for anyone to begin an article about the company by explaining “what the word ‘EA’ in ‘EA Games’’ stands for.” (Notice how ‘EA’ is described as a “word,” not an acronym.)

(What’s the moral of this somewhat obtuse and certainly gratuitous analogy? Perhaps that you shouldn’t try to use KFC in analogies. They tend to drag on for a bit, as you may or may not have noticed while reading the previous two paragraphs. I certainly learned my lesson, so let’s move on to the present day shall we?)

It must be said that things have gotten better at EA since John Riccitello was made president in 2007. A recent profile of the company in the October issue of Edge details much of what has gone right under Riccitello’s reign. First, he acknowledged that the company had grown too big and was releasing too many titles. He laid off staff and trimmed the release schedule. EA COO John Schappert is quoted saying that “A couple of years ago we shipped 67 titles; this year we’ll ship 36. Our goal is: ‘Let’s make sure the titles we’re going to make are great.’”

Under Riccitello, EA expanded the EA Partners program and allegedly made  efforts to improve the creative environment for the company’s in-house developers. By the end of 2007, EA also bought Bioware and made Bioware co-founder Ray Muzyka a Senior Vice President of EA’s RPG division. The long-term effects of this acquisition remain to be seen, however: it could improve the quality of EA’s overall RPG output, but it could just as easily result in a less focused and creative environment for Bioware’s own designers. Still, from EA’s point of view, this was a good step towards creating a more developer-friendly environment within the company (at least as far as RPGs go).

But in spite of these efforts, EA still has a lot to prove to gamers if it wants to become a respectable publisher once more. And make no mistake: “respectability” is the most that a publisher its size can ever hope to achieve. A publicly traded publisher like EA cannot hope to be “loved” or admired by gamers or developers.  Such adulation is reserved for studios (and the occasional first-party developer, e.g. Nintendo). This is because gamers recognize that publishers are not in the business of creating games, they’re in the business of making money. As such, their loyalties ultimately reside with shareholders, not gamers. Sure, EA wants gamers to be happy–and they spend quite a bit of money trying to figure out what gamers want–but gamers are simply a means to achieving the return-on-investment that shareholders expect from the company. Riccitello, after all, was not brought to EA in order to rekindle its creative spirit, but rather to sell more games and help the company regain its once dominant position in the industry (they were putting out mediocre titles long before its sales began to flounder, that’s for sure).

To that end, Riccitello has sought to diversify the company even as he tries to improve the quality of its “core” games. They have made serious in-roads into the casual games market and continue to experiment with “free-to-play” titles like Battlefield Heroes. These initiatives may very well be necessary for EA to keep pace with changes in the industry brought on by the internet, but they also serve EA’s long-term financial interests for slightly different reasons: namely, they will make the company less dependent on the sort of “core” gamer who takes games seriously, pays attention to reviews, and complains loudly when a game fails to meet her expectations. By targeting the lowest common denominator, EA (and most other major publishers) is building a mass audience that doesn’t know much about videogames, doesn’t read game reviews and, most importantly, doesn’t expect their games to be more than a 10 minute distraction to help them pass the time at airports and doctor’s offices. Unlike traditional gamers, these people are not asking for a five course meal, and they  certainly won’t get critical if you overcook the meat. All they want is a bite-sized piece of digital chocolate to get them through the day, and that’s a much easier business to manage.

Allow me to illustrate this point with one last quote from Edge’s recent profile of EA:

Battlefield Heroes attracted mediocre reviews and was heavily criticised earlier this year when the payment model was changed, making it almost impossible to progress through the game without paying for new weapons. “The perception was:  ‘Oh, EA has fucked this up, we’re never going to play again,’” says Patrick Soderlund, SVP and group general manager of EA Games’ FPS and driving titles. “But funnily enough, when we changed the way you pay, we had more players, and the game is now profitable.”

Funnily enough indeed. Hilarious in fact. But like many great jokes, there is a sad truth beneath the laughter. The truth is that EA did fuck up. It released a pretty bad game, then made it less of a game by charging real world money for the privilege of getting better at it. Somehow, this change for the worse attracted an audience and now the game is profitable, which is all they cared about to begin with. Hence the dark, pathetic laughter at EA. Note to Battlefield Heroes players: EA is not laughing with you, they’re laughing at you.

Not that there is anything intrinsically wrong with this attitude. After all, it’s their job, indeed their ‘responsibility,’ to make a profit for shareholders, and to do it as efficiently and painlessly as possible. And like any cunning politician, EA knows that the best way to achieve this is to alienate as few people as possible.  This the sort of corporate attitude that led to EA’s recent decision to eliminate playable Taliban characters in Medal of Honor (a cowardly move that is problematic for several reasons, which Ian Bogost already analyzed quite brilliantly in this essay). As a general rule, then, this strategy requires videogame publishers to pander to the lowest common denominator while simultaneously pretending to care about “taking the medium to another level,” in order to make the game seem less threatening  to casual gamers without alienating the traditional World War II ‘modern warfare’ audience.

Again, my point here is not to single out EA or vilify videogame publishers in general. I simply wish to note the basic fact (obvious to everyone in the music or film industries, but surprisingly absent from many gaming discussions) that videogame publishers are fundamentally different from both videogame developers and serious videogamers. Whereas we believe that a videogame is an end in itself, publishers use it as a means to profitability. This makes them a completely different animal: their priorities are different, their expectations from games are different, and their outlook of “success” will also be different most of the time. Not evil, just different in ways that often run counter to the interests of the medium.

This brings us, finally, to the issue of player tracking. As Bioware’s publisher and parent company, EA’s interpretation of the data collected for Mass Effect 2 is ultimately the only one that matters. So how will it read this data, and to what end? How will their interpretation differ from Bioware’s?

 

‘We deal with numbers’

Did you know that the videogame industry already has a system in place that mathematically determines the quality of every new release? It’s called “Metacritic,” perhaps you’ve heard of it? Like player tracking, Metacritic reduces incredibly complex subjective experiences into numerical values. But unlike player tracking, Metacritic does not derive its numbers from isolated in-game “events.” In fact, Metacritic scores are not really “derived” from anywhere – they are assigned by the site’s staff, whose  impressive qualifications include reading “a lot of reviews.”

You know how it works: the site monitors a wide variety of gaming publications, (including sources as diverse as the New York Times, Eurogamer Italy, Eurogamer Spain, and Eurogamer Plain), reads their reviews for you, and then provides you with a convenient numerical summary of what each reviewer thought, on a scale of 0 to 100. It doesn’t matter if you grade games on a 100 point scale, or even if you don’t grade games at all: as long as you call it a ‘review,’ Metacritic will attempt to assign a numerical value to it. Yes, Metacritic will literally convert another publication’s words into a 100 point scale, even if the publication makes a conscious decision to exclude grades from its reviews. It then “weighs” averages these scores in order to produce a “metascore,” a number that purports to reflect the overall ‘quality’ of a work without resorting to silly “qualitative concepts like art and emotion.”

If only all of life were like that!” says the Metacritic website, surely echoing the sentiments of socially awkward statisticians the world over. Luckily, life isn’t ‘like that’ – it’s just too rich and complex to be reduced to a number. The same goes for videogames – but try telling that to publishers like EA, who now rely on the site to evaluate the output of its development teams. As a developer told Michael Abbott recently, a particularly low Metacritic score means “people lose their jobs.”

The problems with using Metacritic as some kind of arbiter between publishers and developers are well-known by now, so I won’t bore you with more details. The bigger and more complex question is the one first posed by Stephen Totilo in 2008: namely, “why would a development studio ever tolerate publishers setting up deals like that?” Why indeed. Why would a developer ever agree to risk their very livelihood on Metacritic’s subjective impression of a ‘critical consensus’ which, by definition, consists of nothing more than an aggregate of the various individual subjectivities monitored on the site?

While it is true that most smaller studios have no choice on the matter, part of the blame must be placed on the development community itself,  for acquiescing to such a draconian system in the first place. Indeed, many developers direct their anger at individual critics (for depressing their metascore) while remaining deeply ambivalent towards the system that put them there in the first place. Others, like designer Soren Johnson (incidentally, a very talented guy who writes the excellent Designer Notes blog), seem to regard Metacritic as a necessary evil of sorts:

What should executives do if they want to objectively raise the quality bar at their companies? They certainly don’t have enough time to play and judge their games for themselves. Even if they did, they would invariably overvalue their own tastes and opinions. Should they instead rely on their own internal play-testers? Trust the word of the developers? Simply listen to the market? I’ve been in the industry for ten years now, and when I started, the only objective measuring stick we had for “quality” was sales. Is that really what we want to return to?

 

 

I’ve argued before that it is impossible to objectively determine (much less raise) the quality of a product that can only be experienced subjectively. Even Johnson seems to recognize this when he notes that publishers who play their own games would “invariably overvalue their own tastes and opinions.” Well, duh: of course our own tastes and opinions will be central to any activity that requires us to taste and opine – that’s just common sense. Still, why exactly would this be a problem? Publishers are the ones financing the product after all; why, then, shouldn’t it reflect their taste? By Johnson’s logic, we should also worry about a restaurant owner who hires a chef “just because” he enjoys the person’s cooking.

(His last point – that sales are not a good way to measure quality – is more compelling, and I’m certainly not one to argue that against it. But it does have one thing going for it: actual objectivity. True, sales tracking may not be a reliably objective measure of a game’s quality, but at least it’s an objective measure of something, which is more than we can say for metascores.)

 

I don’t want to keep harping on about the role that developers have played in allowing such a flawed system to be used against them because, as I mentioned earlier, most simply don’t have a choice. If forced to choose between placing their royalties at the mercy of Metacritic or not making the game at all, most  studios will understandably go with the former option. In this respect, they’re like the proverbial starving musician who signs a record deal without reading the contract only to discover, years later, that she was screwed by the label.

So perhaps we should have addressed Totilo’s question to publisher’s instead. Why do publishers rely so much on metacritic anyway? What is it about the site that publishers find so attractive and useful when dealing with studios? Here, Johnson gives us a hint, when he notes that publishers “don’t have enough time to play and judge their games for themselves.” But it’s not just that they don’t have the time, it’s that they don’t have the skill or the know-how to play and pass judgment on games. See, executives are numbers people. They like and respect numbers, and have very little patience for the ambiguities of art, language, and criticism (this is also why many business leaders are so contemptuous of  “fancy political rhetoric”).

Thus, Metacritic seems like an ideal solution to many publishers: it reduces numerous qualitative opinions into a single number and, since the number is determined by an outside party with its own (secretive) methodology, it confers the illusion of objectivity upon the final number. But don’t take my word for it, here’s game industry marketer Bruce Everiss (itallics mine):

I have used [Game Rankings] countless times as a tool to help in my work. Most notably to prove to the directors of Codemasters that their game quality was slipping in comparison to their direct competitors.

Then in 2001 Metacritic came along and changed the world. Firstly they convert all the review scores into percentages, then they average them to come up with one figure. (They also weight the average so more respected reviewers have more influence.) This single figure to represent a game is a very powerful thing and everybody in the industry is far more aware now of game Metacritics than they ever were of individual review scores, they have become the standard benchmark for the industry.

Of course, the only way that a site like Game Rankings or Metacritic can actually “prove” anything is if we buy into their methodologies, and since Metacritic keeps its methodology (including its “weight” system) a secret, it seems odd that Everiss would use those numbers to prove anything. But we’ve already been over that. The point is that those numbers are being used as if they were the final word on a game’s quality, even though their reliability is questionable at best. In short, Metacritic scores have empowered publishers to make decisions over “quality” in spite of knowing that they lack the gaming knowledge to do so, and they do this by drawing specific conclusions from numbers that at best provide us with nothing more than one person’s impressionistic assessment of a critical “consensus.”

Seeing as most developers actually take the time to read individual reviews of their games–and therefore better suited to put Metacritic scores in their proper context–it is hard to see how this publisher-dominated metascore system benefits anyone other than the publishers themselves. At the very least, it has empowered publishers, by giving them a greater say over areas of development that typically belong to developers.

My fear is that player tracking will eventually create the same situation: i.e., statistics that were meant to aid developers when starting work on their next game may be taken out of context by publishers, who would then use these numbers as “proof” of what audiences want from future games. The result will be even less risk-taking than we currently see in the games industry. The era of focus-group-tested games will slowly give way to an age of mathematically tailored experiences, targeting the lowest common denominator with unprecedented precision.

Personally, that’s not a future I want to see – but then again, I’m not a numbers guy.

GamePub Story

Let’s pretend for a second that I am an EA executive. It’s my job to supervise an external development team that is currently working on a Mass Effect spin-off project scheduled to be released before Bioware finishes work on Mass Effect 3. Being a responsible executive, I’ve decided to do some research before my next formal meeting with the studio in charge of the spin-off. Of course, this research doesn’t involve me actually playing the game–hell no. By research I mean reading up on Metacritic scores, development schedules, sales numbers, and – you guessed it – Mass Effect 2 player data.

 

Some of the data really startles me. I focus my attention on two in particular:

  • 80% of players played as a male Shepard?! (I wonder how much time and money was spent developing the female Shepard’s character models, dialogue options, and voice-acting).”
  • 80% of players chose to play as a Soldier? More than every other class combined?! (Was our investment to develop the 5 other classes worth the other 20%? And of that 20%, how many of them were belong to the 50% of hardcore fans who imported their saves from the first game?).”

So I jot those stats down on my Blackberry and head out to meet the developers in person.

When I get there, the news is not good. The project manager tells me that the team is six months behind schedule and significantly over budget. “This is our first attempt to make a game within the Mass Effect universe, but we’re confident that our next ME spin-off will take less time to complete,” they explain. “All we need is an additional six to ten months to deliver a product worthy of the Mass Effect brand.”

But I don’t want to hear that. Having reviewed my company’s release schedule prior to the meeting, I know that Mass Effect 3 is scheduled to release in late 2012 or early 2013, and since the whole point of releasing this spin-off is to satiate the fans while they wait for the third installment of the main series, we simply can’t afford to push it back another six months. “Sorry,” I tell them, “the game simply must be released in 2011–we’re going to have to figure out a way to make this work. I could perhaps get you some more funding, but we also need to figure out how to cut costs on your end–otherwise, our request for more money won’t be too well-received at corporate headquarters.” This prompts the creatives sitting at the table to roll their eyes at me.

“So then….what do you propose we cut? Any ideas?,” asks one of them with a hefty dose of sarcasm.

“As a matter of fact, I have some suggestions right here on my Blackberry. For instance, how much time and money would we save if we axed the option of playing a female character altogether?”

“Well that would certainly help us quite a bit, though I’m not sure it’s enough; besides, giving you a choice of gender is a Bioware tradition, and we want to do them justice with this game.”

“Of course we do,” I answer, “we want to be as faithful to the series as possible, but remember that this game will not form part of the main series; since it is only a side story, I think we can persuade the guys at Bioware to let us limit the game to a male protagonist just this once, especially if you give him a compelling back-story.” After showing them the Mass Effect 2 player statistics and some further discussion, I manage to convince them that this is a good idea, a task that was probably made easier by the fact that–surprise plot-twist ahead!–there were no women present at the meeting. But that is still not enough to bring development back on track, so I move on to my next idea, which proves to be far more controversial.

“What!?!?! You want us to axe every other class in the game!? You really want us to limit players to the role of soldier??? That is simply unacceptable,” says a visibly angry lead designer. “No. That’s just not happening, and I don’t care what statistic you show me…the class system is part of the game’s legacy–hardcore gamers expect this from us, there is just no way we can risk alienating them like this. Trust me, you’ll get a backlash from the dedicated fans, and that won’t be good for any of us.”

“Besides,” adds the project manager, “we’re already pretty far along in the design of the various classes, so that wouldn’t really cut costs as much as you’d think.”

“But would it cut development time?” I ask.

“Yes, maybe. But we’d still need additional funding, so it would be a waste of resources to simply abandon something that our team has been working on for months.” Faced with this impasse, I lean back on my chair and close my eyes for a second. The room stays silent until, suddenly, a little light bulb goes off in my head.

“What about this,” I tell them. “What if we save the other classes for DLC? That way, we can postpone developing them for the time being, get additional ‘DLC funding’ to finish the classes at a later date, while earning additional income from the hardcore players, who are the only ones interested in playing with them in the first place.” Once I show them that a whopping 80% of players chose to play as a soldier in ME2, and that our company has committed itself to prioritizing the hardcore-gamer-cash-cow that is DLC, the  team grudgingly goes along with my brilliant idea. And so the meeting comes to a close, I express my gratitude towards the development team, and assure them that “my bosses at EA will be very grateful for your understanding, and grateful to know that we already have promising DLC content in development!”

Now back to work guys….”

Visibility is a trap

Think of this as a kind of Nietzschean parable: the point is not so much to provide a faithful account of the future, as it is to speculate and warn about what could happen if we continue down this path. As we have seen, publishers and developers have profoundly different ways of looking at the world, and this creates the possibility of conflict when it comes to interpreting player data. Developers may look at a statistic such as “80% percent of players chose the soldier” and see it as an eminently solvable problem of menu design and presentation, but publishers could just as easily seize on that as a justification to cut costs or–worse–to make additional money off of the dedicated fan who  is willing to pay for DLC.  The worst case scenario is that developers will end up losing such arguments more often than not, and we the audience will end up settling for lesser games.

Trust me, dear friends and developers, I get why you would be excited by the prospect of using new technology to learn more about your audience. Why wouldn’t you be?  But please be careful how you collect such data; be careful who you share it with; and for goodness sake, be ready to defend your findings in front of the people who pay your bills, lest you end up in another meta-prison of your own making.

In short, beware of unintended consequences, and always remember Foucault’s prophetic warning: “visibility is a trap.”

About these ads
This entry was published on December 7, 2010 at 1:28 pm. It’s filed under game culture, game design, videogames and tagged , , , , , , , , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post.

23 thoughts on “Tyranny of the Masses

  1. The real problem with the Mass Effect statistics is that they don’t result from a controlled study, and so they might not mean what they say. It could be that 80% of players choose Marcus Fenix Shepard with facial modification because they genuinely don’t care about gender or class choices. It could be that these are the default options on the menu, and appear in all the marketing materials. It could be that the Soldier has notable gameplay advantages over the unpopular Engineer, and Mass Effect’s 80% male userbase prefers to choose an avatar of their own gender on average. EA is unfortunately unlikely to consider any of these other options and just cut Jennifer Hale instead.

    • @Robyrt

      Yup, I think those are all possible causes. Let me add another one to that list: aside from the gameplay advantages that you mentioned, the soldier is (at least from a gamer’s point of view), the only class with an instantly recognizable name and function. Everyone knows what a soldier does, but what the hell is a Biotic? Or an Infiltrator? Of course, these classes don’t stray too far from the traditional offerings of Western RPGs (a Biotic is essentially a mage and an Infiltrator is comparable to “rogues” or stealth specialists), it’s just that their names make them seem far more esoteric than they actually are. Thus, some Mass Effect newcomers who would normally choose to play as a mage or a stealth specialist may have been misled into thinking that those options were not available to them.

      Honestly, I think the overwhelming popularity of the soldier class is the result of a wide variety of factors–the ones you mentioned + several others that we probably haven’t thought of (how else could we explain the 80%? That’s just an absurd number!!!).

      Hence it is not at all clear what (if anything) the data is actually telling us about the game. Whatever lessons we take away from these stats ultimately depends on how we interpret them…and as you said, EA is unlikely to take a nuanced look at the subject.

      Anyway, good points–thanks for the input!

  2. look I generally agree that metrics are not the 100% solution for any given problem, but I’m not entirely convinced people are literally arguing “We need metrics for everything, we need to share all of that data publicly and with everyone, and we need to make every single important decision based solely off metrics”. It’s a strawman and totally not how metrics work. You can go ask someone who works for Zynga how they use metrics and they will say “Oh we use it to get some basic information and do basic a/b testing but if something feels ‘wrong’ to us as a game designer, we won’t do it no matter what the metrics say” (I listened to the guy who does Mafia Wars give example – an “instant win” button would have GREAT metrics, but obviously that would be a terrible idea. And metrics also helped him identify the fact that players would stop playing as soon as they hit a hard requirement for more friends in their gang in order to continue – which they fixed). In fact, I’ve never seen anyone say “Well it completely destroyed our game but the metrics said to do it so we did”. I’ve seen executives say “This isn’t making any money, so we’re cutting it” – but I don’t think you can argue that “making money” is a bad use of metrics in a capitalist economy.

    Which leaves the obvious question: If metrics are so bad, how have we survived so long with them? How have we as a society managed to create so many great works of art with “how much money did it make” hanging over our head? And the answer is “sometimes good things make lots of money”, and also “sometimes money can be ignored in pursuit of great things”, which is to say: sometimes good things have great metrics, and sometimes metrics can be ignored in pursuit of great things. I see this community-wide freakout about Bioware using metrics and Zynga using metrics and etc etc etc. despite the fact that this is a solved problem, and a well-known solution to designers working in the industry. it’s like the first thing they teach in usability classes: When a user says something, don’t take it at face value. Dig deeper and understand the feedback at every possible level.

    Metrics are useful. I say this as a designer and as someone who works in software. Without metrics, I would not be able to identify the correct design and argue successfully for it. Metrics give us information about real users in real conditions, ie not sitting in a usability lab getting watched by usability engineers. If you’re going to say “well they have the potential to be used poorly”, you might as well argue against statistics as a tool because it also has the potential for abuse.

    I mean I will come back here and apologize if Mass Effect 3 comes out and there is no female and no class other than soldier and the devs say “well remember those metrics we released, LOL!!!” but I think the much more likely outcome is that EA/Bioware will say “hey if all our players are playing gun-heavy classes, let’s spend more time improving the combat system” (which, let’s be honest: is TERRIBLE, and badly needs the improvements) or improving other classes (let’s be honest 2: engineer is objectively the worst class) as opposed to cutting all the other classes outright and firing jennifer hale and etc etc etc.

    • @zach

      Thanks for the comments Zach. You’re right that statistics are not “new” field, and that metrics of one kind or another have always existed in the games industry. But the type of data collection discussed in this post is actually a very new phenomenon. (It wasn’t used in the first Mass Effect, for instance.) This is why people are talking about it so much. With that out of the way, I’ll try to respond to some of your specific points.

      1) You begin by saying that you “generally agree that metrics are not the 100% solution for any given problem”

      Fine, but before talking about solutions, let me ask you this: what exactly is the problem that needs to be solved, with metrics or otherwise? Has there been an increase in poorly made games? Are gamers really underrepresented in the gaming industry? Were games worse prior to the invention of sites like Metacritic?

      My goal in this post was simply to consider the possible dangers of adopting this form of data collection in an industry that, historically, has been way too eager to adopt various forms of consumer research (with frequently tragic results: remember Bubsy for the SNES?). But I never wanted to suggest that player tracking is uniformly evil or that developers/publishers would be foolish enough to use these metrics at the expense of every other metric at their disposal. Very few things are 100% positive or negative…player tracking, like almost anything else, will probably have good qualities and bad ones.

      It is true that I mostly focused on the potential negatives. I did this because the upside to these metrics seems self-evident. Of course developers want to know their audience, and of course they want to know how their game is experienced. But the issue at hand is whether this form of data collection actually achieves these goals in a meaningful way, and at what cost.

      To that end, I outlined my view on some of the potential risks…whether these risks are worthwhile really depends on the nature of the problems that player tracking claims to be solving. And with that, I again pose the question to you: exactly what is the problem that these metrics are trying to solve?

      2) “Metrics are useful [in game design].”

      Yes, I agree that metrics can help you make good (or bad) design decisions; but so can playing Tetris or Super Mario Bros. The lessons you draw from Mario may help you create a great game (Meatboy) or a bad game (Bubsy). It all depends on what you take out of it and how you’ve interpreted the game or statistic in question. So give yourself some credit: the data has nothing to do with you being “able to make the correct design.” In fact, it’s the other way around: i.e., your use and interpretation of that data is relies heavily on your preexisting perspective on game design. Without that context, the data would be meaningless.

      In other words, the “meaning” of any given statistic is always subject to interpretation and, more often than not, this interpretation will be filtered through our own preexisting beliefs and perceived needs. For instance, we can all agree with the basic fact that Obama’s low approval numbers in recent months pose a problem for his administration,  but the way in which we interpret the causes and implications behind his drop in popularity tends to be heavily influenced by our politics. If you’re a Republican tea-bagger, chances are that you interpret this drop as “proof” that the general public has “finally caught up” with what you’ve been saying all along: namely, that Obama is a radical leftist weirdo or, at the very least, ‘taking the country in the wrong direction.’ If you are an Obama supporter, however, then you may blame the low poll numbers on a ‘communications failure’ on the part of the Whitehouse, or rationalize his decline as a cyclical phenomenon that affects almost every president during their second year in office.

      In each case, our understanding of the public’s feelings toward Obama is determined by preexisting ideological biases with built-in implications for what Obama (and the public) should do in the future. If you think Obama’s ideas are sound, you’ll place the blame on him personally for failing to communicate them effectively; if you don’t, then you’re likely to treat his personal decline in popularity as a general rejection of the ideas he champions.

      Keep in mind that both sides could be completely honest and still reach these radically different conclusions, which is exactly what I meant when outlining the different perspectives of publishers and developers. I wasn’t saying that publishers would deliberately screw things up because they’re “greedy,” only that they would interpret the data according to their own needs and beliefs, which don’t always coincide with developers.

      This is why I made a point of placing my argument within the developer’s camp; I never pretended to be an unbiased observer; I never declared the publisher’s position to be “unjust.” All I said was that, from a developer’s point of view, this type of out-of-context data collection is risky, because each side is likely to interpret the data differently, and because publishers always have the advantage in these kinds of debates.

      3) “I mean I will come back here and apologize if Mass Effect 3 comes out and there is no female and no class other than soldier”

      That won’t be necessary because it won’t happen. The Mass Effect team is too important for EA to risk alienating them like that. They don’t want to mimic what happened between Infinity Ward and Activision. Besides, they’ve already promised to allow the importation of save files from previous games, and they definitely don’t want to risk a gamer backlash by switching course now. That’s why I used the example of a Mass Effect spin-off, and why I imagined a scenario in which development was being outsourced to a studio with “something to prove.”

      Still, it may very well be the case that none of this ever happens at EA. Maybe they’ll surprise me. That would be fantastic. But given that many publishers have no problem with using metascores to determine developer compensation, it seems naive to say that this data will never lead to a similar situation. Except this time it could be worse, because there is a lot more data to collect, and all of it is directly tied to specific aspects of the game playing experience.

      Feel free to let me know if I misread your views…I’m always down for more discussion ;)

  3. It might be new for console games, but if you look at any Microsoft product from the past 10 years or so you’ll notice you have the option to “opt-in to detailed reporting” in several products, including Messenger, Security Essentials, and Windows (“Send error report”). It’s not new in software, and software engineers use metrics to resolve disputes between engineers, to allocate resources for future versions (identifying troublesome areas that need more work), and to basically make sure that all the bases are covered in the event of a disaster (knowing the second a service goes down). it’s a fact of life you never have all the resources you need in software development. if 80% of my users weren’t seeing any of my other content and I needed to make a decision in order to launch on time, you bet I would cut that 20% right out if it would save my project – although at that point the project is probably fucked anyway due to poor planning. That’s not a “bad” thing, that’s just the reality of project planning, and it happens with or without metrics.

    Games are really, really shitty pieces of software. They crash, they need specific drivers, they can’t handle hardware combinations, they aren’t performant, they have terrible UI. And yes, I would argue that as game rules get more complex, the software containing those rules aren’t always able to keep up – although you pull a neat trick by saying metrics can’t be objective, but we can somehow agree if games have or have not gotten “better” or “worse”. (I can find at least one dude who thinks Bubsy 3d is the greatest game ever made, and I personally didn’t enjoy Super Meat Boy in the slightest – and i think the 50 million people who play the metrics poster-child farmville, primarily made of entirely new demographics who never played videogames before, would think this is certainly a golden age of gaming)

    There is no magic “design intuition” that would ever let you consistently spot all those problems metrics find. You can only put it in the hands of as many users as possible to get as many data points as possible. And once you get those results, there is certainly no magic “design persuasion” power that could convince a bunch of your type-A data-oriented coworkers that there’s a problem with THEIR code/design/interface, without some pretty solid numbers to back it up. NOBODY thinks metrics are some objective truth handed down from on high (once again, that’s the strawman argument), and you can bet your ass that the Bioware guys are trying just as hard to understand the reason 80% of people play Soldiers as you are, because they all have varying degrees of investment in that statistic.The guys who worked on the Soldier class are probably bragging while the Engineer guys are insisting it’s a problem with how the metrics collect data. And without those metrics, Bioware wouldn’t have any incentive whatsoever to improve the soldier experience (dumb, boring, repetitive) and to try to find ways to convince players to look for other experiences besides Soldier (for replay value, to accomodate larger audiences, whatever). Not to mention knowing the exact number of people who bought the game new and used the DLC codes versus the number who bought used and purchased the DLC by hand versus the number who never bothered with any sort of DLC whatsoever has incredible implications for EA’s “$10 Project”…

    As a community, it’s kind of important to recognize that metrics reveal things about ourselves: Nobody plays the single player campaign in CoD to completion (important when we’re all arguing if “No Russian” crossed a line and it turns out only 1% of the people who play the game actually got that far). There’s an almost perfectly even distribution of race preference in Starcraft 2, except at higher levels where Terrans are slightly underrepresented (balance & metagaming are 2 incredibly important parts of online play). 100% of people who played the Avatar: Last Airbender game got 100% gamerscore (in case you wanted to know how to boost gamerscore). I like knowing what other people in the community do, because it helps me understand the diversity of my experience vs the community at large, it forces me to understand the varied kinds of playstyles in the ecosphere, and it helps me stay relevant to the new kinds of gamers constantly coming on board. I don’t see how any of that is counteracted by a fake conversation, with a fake studio, that you made up on the spot, in order to prove a rhetorical point in lieu of any actual evidence that metrics are harmful. I don’t see how advocating for ignorance of the data is helpful for anything except a purely philosophical stance.

  4. Pingback: The Sunday Papers | Rock, Paper, Shotgun

  5. “Nobody plays the single player campaign in CoD to completion (important when we’re all arguing if “No Russian” crossed a line and it turns out only 1% of the people who play the game actually got that far).”

    I did, and I know various other people who did, so this is wrong. I’m sure your actual point is that the vast majority did not, but that’s kind of the thing; I don’t necessarily have any problems with stat-tracking, but the idea that developers can be slave to the analysis such that outliers literally aren’t even acknowledged is one of the things that potentially worries me about it.

  6. IncredibleBulk92 on said:

    This is actually something that I starting thinking about when Bioware started releasing information about Dragon Age 2. Your stuck with a male character who has a very specific background. Oddly similar to the development scenario you posted above.

    However I’m interested in the exact figures of the 2 statistics you posted above. I’m willing to bet a lot of people picked the default Shepard who is in the game’s marketing which is a shame as the 2 other pure breed classes are far more interesting to play. Personally I hope that the only result of these 2 statistics is that Bioware give the soldier class a bit more depth but that may be asking a bit much.

  7. Wow IncredibleBulk92, I had no idea about that Dragon Age 2 info! That is surprising considering the multiple intros in the first game.

    I think you’re right to suspect that the game’s marketing and official screenshots had a lot to do with the dominance of male-soldier Shepard. I’ve never understood why they do this (they did it again in the ME3 reveal!) after spending the time and effort to create a female protagonist. The fact that Jennifer Hale’s voice acting is so good adds insult to injury…

    Mass Effect is not the only series guilty of this….this post over at the Border House criticizes Fable 3 for doing the same thing http://borderhouseblog.com/?p=2321

  8. @Zach

    –> “It might be new for console games, but if you look at any Microsoft product from the past 10 years or so you’ll notice you have the option to “opt-in to detailed reporting” in several products, including Messenger, Security Essentials, and Windows (“Send error report”). ”

    True, but that’s exactly my point: all of those things are just functional tools designed to achieve specific things: e.g., it makes sense for Messenger to keep track of how users communicate through it because allowing us to communicate is the entire point of its existence! But videogames don’t have such a clear-cut purpose. Games can be a mode of expression, entertainment, art, all of the above, etc. Since these are all subjective aspirations, there is no way to empirically determine whether a game has achieved them.

    I have no problem with tracking data geared towards the technical side of game development. If the point is simply to uncover and report bugs then I’m fine with it….my problem is with collecting data from in-game events. These events are always experienced subjectively, and are the result of Bioware’s (also subjective) design choices.

    Speaking of which:

    –> “you pull a neat trick by saying metrics can’t be objective, but we can somehow agree if games have or have not gotten “better” or “worse”.”

    No trick intended! Of course people can disagree with my take on Meatboy and Bubsy. They’re games, after all, not tools or operating systems!

    I was actually talking about Bubsy for the SNES, which was bland but at least playable. Didn’t mention Bubsy 3D because that game was so broken on a technical level that using it felt like creating a straw man. Still, people have the right to like and defend that game as well….who cares if Seanbaby called it the 17th Worst Game of All Time ( http://www.seanbaby.com/nes/egm17.htm ).

    –> I don’t believe in “magic design intuition,” but I do believe in earthly design intuition. You don’t?

    –> “NOBODY thinks metrics are some objective truth handed down from on high (once again, that’s the strawman argument)”

    Here is EA’s president talking about Metacritic: “Our core game titles are accurately measured and summarized by these assessments, and that is a very big deal.” http://www.escapistmagazine.com/forums/read/7.54796-EA-CEO-Upset-by-Poor-Review-Scores

    –> The growth of stats tracking is all but inevitable, I’m perfectly aware of that. Which is why I was advocating a careful, critical approach in the way we interpret it. Statistical analysis is fine, but not when it discourages creative risks. As Mman said, we must guard against becoming ‘slaves to analysis,’ whether it comes from Metacritic or in-game statistics.

    One final note: making the data public is better than collecting it and keeping it a secret, as at least the latter allows us to challenge its ostensible meaning and expose its blind spots (which is what Robyrt did in his comment.) So kudos on that front to Bioware.

    P.S., Can you please tell those $50 million Farmville players to stop spamming me?

  9. Himself on said:

    What zach said: in the example you gave the imaginary studio had a choice between cutting out the content that only 20% of players will see and, say, not releasing the game at all.

  10. Pingback: קישורים שנשארים | המשחקיה

  11. Simply wish to say your article is as astounding. The clarity to
    your publish is simply nice and that i can assume you are knowledgeable on this subject.

    Well along with your permission let me to clutch your RSS feed to
    stay updated with impending post. Thanks one million and please continue the enjoyable work.

  12. Hmm it appears like your site ate my first comment (it was extremely long) so I guess I’ll just sum it
    up what I wrote and say, I’m thoroughly enjoying your blog.
    I as well am an aspiring blog blogger but I’m still new to the whole thing.
    Do you have any recommendations for inexperienced blog writers?
    I’d genuinely appreciate it.

  13. Also can the simpsons drawn on out cheats previously have an impact on politics.

  14. I was recommended this website by my cousin. I am not sure
    whether this post is written by him as no one else know such
    detailed about my difficulty. You’re wonderful! Thanks!

  15. Tremendous things here. I am very glad to look your
    article. Thank you so much and I am looking ahead to contact
    you. Will you please drop me a e-mail?

  16. However, if the delinquent is that the stomach learn B12 vitamins, it may be necessary
    to get B12 shots, which the directly into the veins.
    Are Dietary Supplements as Effective as Natural Nutrients.
    If you’re older, or a bit overweight or obese,
    your body is naturally less efficient at making vitamin D from sunlight.

  17. I’ll right away take hold of your rss as I can’t find your email subscription link or newsletter service.
    Do you have any? Please permit me understand so that I may just subscribe.
    Thanks.

  18. First off I want to say wonderful blog! I had a quick question in which I’d like to ask if you don’t mind.
    I was curious to know how you center yourself and clear your head
    before writing. I’ve had difficulty clearing my mind in getting my thoughts out.
    I do enjoy writing however it just seems like the first 10
    to 15 minutes are lost simply just trying
    to figure out how to begin. Any suggestions or tips? Cheers!

  19. Thanks for sharing your info. I truly appreciate your efforts and I am waiting for your
    further post thank you once again.

  20. It’s wonderful that you are getting thoughts from this article as well as from our discussion made here.

  21. But they usually beat around the bushes and expect you to do what Arnold does, a million sets and 5 hours of steady
    pumping iron. To be able to be in a kickboxing class in Elmhurst,
    NY and know that you are accomplishing many different aspects of life is just amazing.
    Article Source: Mc – Kay has worked with the fitness industry for over 20 years now.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: