10 December 2011

How Effective Are Games at Training

Michael Peck asks a very good question in a new TSJ article.

“Games have exploded, without a lot of analysis or forethought,” said Leslie Dubow, project director for Games for Training at the U.S. Army Program Executive Office for Simulation, Training and Instrumentation (PEO STRI). “[There’s] not a lot of opportunity for it. We’re a nation at war.”
Military training and simulation experts don’t try to hide the uncertainty surrounding levels of gaming effectiveness. Nor are they apathetic about the issue. It is more a tacit acceptance that this is the nature of games.
It would neither be practical nor cost-effective to evaluate games, Dubow said.
“Our strategy is that these are throwaway products,” she said. “Five years is not a long time to invest in something. By the time we have something accredited, we will be going on to the next thing. The research associated with the validation of training effectiveness is not inexpensive. We are not funded at our level to do it, or have the time.”
Indeed, assessing the effectiveness of games is difficult. It’s easy to determine whether Rifle A is better than Rifle B based upon how many rounds hit the target. But how do you measure the effectiveness of a cognitive counterinsurgency training game like the Army’s “Urbansim”?
Robert Bowen, chief of U.S. Army Training and Doctrine Command Capability Manager - Gaming (TCM-Gaming), which oversees the Games for Training program, points to two issues that make it a challenge to evaluate games like the “Virtual Battlespace 2” (VBS2) first-person shooter: the use of games in collective training, which means assessing effectiveness in a group rather than an individual context, and the cognitive nature of games.
“It’s very hard to put a quantitative kind of substance to a collective training environment,” Bowen said. “It’s easy to measure marksmanship. When you do collective or cognitive, it gets very, very hard to measure that.”

And you know what? I was working on this exact research at Ohio State before they fucked me and disenrolled me. They could've been pointing to the research of one of their own grad students as a success story on this, but they let their control-freak attitudes get in the way. Ah well, fuck'em.

By: Brant


Guardian said...

From the quote above, "Five years is not a long time to invest in something. By the time we have something accredited, we will be going on to the next thing," according to Dubow.

1) Fix your accreditation process. It is a bureaucratic nightmare and a jobs program for so-called Information Assurance experts who wouldn't know the difference between a SQL injection attack and a cross-site scripting vulnerability.

2) Welcome to reality. In the real world, successful enterprises are leaning forward in the saddle, planning and designing the Next Big Thing before the Last Big Thing has even been released. Steve Jobs died the day after the iPhone 4S was released. He was working on the iPhone 5 (rumored to be released next year) and a five-year roadmap for Apple's products. DICE, the developers behind Battlefield 3 (the best of the mass-market tactical action games), have been working on expansion packs since before the game itself was released and I would imagine that some of them have next-gen console prototypes from Microsoft and Sony in a room running some amazing tech for Battlefield 4. Heck, we have a five-year product roadmap at my little startup and our first product hasn't even been released yet.

-- Guardian

Dan Eastwood said...

Sometimes Graduate school is about politics and institutionalized hazing. I have no doubt they screwed you out of a degree and a lot of money. That really sucks, but the knowledge and ideas are still yours. A PhD qualifies you for ONE job - teaching at a college or university. There is nothing stopping you from taking your ideas to and doing something with them.

On the other hand, it's harder to get paid for that outside of academia. I'm working on a book myself, but not making much progress.