David Zheng

On the third floor of the Yale Child Study Center, a white-walled room hides behind the receptionist desk. The room is small: a round folding table, a few chairs, a refrigerator in the corner, a scale, a collection of loose papers lying on a wooden desk (instructions for making a puzzle, a list of patients). On the east wall, a rectangle is marked out in tape on the carpeted floor. Tess Anderson, a postgraduate associate, watched me as I stepped into the center. It was 9 a.m. The office was empty. I was there to play a game.

“We ask people to not move out of the taped section,” Anderson said as she fiddled with the computer that was hooked up to a TV monitor in front of me. Input logs flashed and disappeared. Though she would usually read the game’s instructions aloud to patients in the center — in this room, mothers and their 7 to 12-year-old children — I was an exception; I already knew the rules.

The game design was simple. Still a prototype, it evoked Wii Sports of yore — all back-and-forth movement, exertion and strategy, strategies for avoiding exertion. The goal in the first round was to catch falling rocket ships on a launch pad. In the second, it was tennis balls on tennis rackets. The launch pad-turned-racket hovered above my head on the screen, my body silhouetted in clumsy video-capture, swerving wildly as the objects fell. Images cluttered the left and right of the screen: a spider and a starfish; a spider and a flower; then, later, a scrunched-up angry face and a neutral face.

After I finished both rounds, Anderson shut down the computer. Normally, she explained, mothers and children fill out a questionnaire regarding their anxiety before and after playing (for example, ranking “I feel upset” on a scale of one to four).

Sweaty and tired, I had almost forgotten that the falling rockets and tennis balls and where they were caught (near the spider? near the flower?) hinted at something clinically significant: levels of avoidance, a key anxiety measure. If I were a patient, an Excel file would reveal each time I turned toward and away from the spider, spiderweb or angry face, accurate to the 30th of a second.

The monitor hummed as I stepped out of the taped rectangle, game finished. A jumble of wires complicated the computer shutdown, though the game had recently upgraded its interface. The technology was still fussy, as though it didn’t yet know it had a job to do.

—

Interrogating reality

Yale Interactive Kinect Environment Software, or YIKES, was developed by Eli Lebowitz of the Yale Child Study Center, in conjunction with autism-focused researcher Frederick Shic GRD ’08 and the rapid prototyping company PreviewLabs. Lebowitz believes that both avoidance and approach — how motivated a subject is to achieve a goal in spite of anxiety — are key elements in anxiety research. In a typical study, a researcher would ask a patient with a spider phobia how close they were willing to get to a real-life spider. YIKES removes the middleman.

“Anytime you as a doctor or researcher ask somebody to do something, you’re placing a strong demand on them,” Lebowitz said. “Their motivation is going to be to try to do the thing that you ask and the behavior might be different than what they would do otherwise.” YIKES provides a new method of studying “natural” behavior, how anxiety subconsciously manifests when the subject is focused on falling rockets. Intent on the game, subjects should be motivated by their score alone. (Ten points per rocket or tennis ball. I couldn’t get past 2,100).

In a 2014 clinical study, Lebowitz’s lab observed 86 children with demonstrated anxiety, as well as their mothers, playing YIKES. The goal was to determine whether parental avoidance would point to avoidance in children — spiders, a common phobia, were the test case. Not only did the study determine that children subconsciously modeled their mother’s avoidance, but the results were statistically significant enough to unofficially deem YIKES an accurate means of measurement — “another tool to include in a toolbox of ways you can address the question you want to,” according to Shic.

Shic, whose lab at Yale used the versatile YIKES interface to measure and detect autistic traits, is uniquely equipped to understand the futurity of such a toolbox. Currently an associate professor in the University of Washington’s Department of Pediatrics, he started his career as a game developer for Sony PlayStation before beginning to program in a friend’s magnetic resonance spectroscopy lab. He shifted his focus to autism research after he obtained a Ph.D. in computer science.

“Video games pay a lot more,” he said, laughing. “[But] at some point I realized that if I’m going to sleep under my desk, I should do it for something I find value in. I just loved asking questions.”

Fifteen years ago, a discussion of game technology was a discussion of its potential harms: Did violent video games yield real-life violence? Were they an art form or a commercial evil, innately dangerous to children?

Now, Shic believes that game technology — motion tracking, virtual reality and augmented reality, all often referred to under the umbrella term of “blended reality” — exists in what he terms an “incredibly nuanced gray area.” Questions are complicated, even unanswerable, addressing not the potential harm but rather the potential good of an ever-evolving medium.

Yet as technology finds its way into mainstream medical research and education at Yale and its peer institutions, ineffable questions demand concrete answers: whether blended reality is vital or disposable, whether it will alter medical research or be merely a footnote.

It’s uncharted territory — researchers are gambling on the technology’s longevity. Shic, for his part, is optimistic; the fundamental principles of research stand firm, in our reality as well as in virtual ones.

“The development of games tends to be the iterative development of scientific questions,” he said. “One question leads to the next.”

—

“It’s a game. You can experiment.”

While blended reality is a recent addition to labs and patient logs, the technology had been a pillar of the virtual landscape long before the development of Facebook’s Oculus Rift or HTC Vive, the current most-cited platforms. The image of a player in a bulky headset, tripping over furniture, is an old one: In the 1980s, Sega’s Master System 3D glasses generated an at-home 3D effect, and Nintendo launched their Virtual Boy headset in 1995.

Video games, not unlike medical research, offer a certain degree of creative freedom via a clear distinction from the real world — it’s what happens in Vegas, if Vegas were built out of pixels. Blended reality, when it works, enhances this freedom, blurring easy distinctions between real and make-believe. “Serious games,” or games meant for a particular, often educational purpose, have slowly entered this growing field, buoyed by increased attention to blended reality.

Bernard François, founder of PreviewLabs, attributes this newfound welcome to a younger generation’s inheritance of game development. PreviewLabs, based in Belgium, recently moved their United States office to Milford, Connecticut. The space is still near-bare, the most colorful element the glow of François’s computer screen. When I asked about games he played as a child, his face lit up as he Googled images and YouTube videos to show me, huffing with impatience as an advertisement hindered his demonstration. Games, serious or not, have allowed him to bend the rules of a world that mimics our own, and this same experimental freedom is inherently imbued in serious games.

“I’m a pacifist,” he said. “But if I have a game and it’s a simulation of a city, I could do reckless driving in a game without doing it in real life. It’s a game. You can experiment.”

YIKES is the brainchild of Lebowitz and Shic but was engineered and prototyped by PreviewLabs. François referred to it as a “deterministic” interface, with a code that drops objects in the same location each time. Yet experimentation is a cornerstone of another PreviewLabs partner, Yale’s Center for Health & Learning Games. François is helping to program smokeSCREEN VR (a working title), a collaboration with the Center’s play4REAL Lab designed to highlight the risks of vaping and teach teenagers refusal skills by allowing them to speak refusals aloud.

smokeSCREEN VR, epitomic of the “serious game” genre, reimagines smokeSCREEN, a 2D computer game developed by the Center’s play2PREVENT Lab in 2017 that focused on nicotine intervention more broadly. Lynn Fiellin, who founded the Center and directs play2PREVENT, emphasized that smokeSCREEN and its VR counterpart are meant as alternatives to typical health curriculums, ways to make mandated lessons deceptively fun. Like YIKES, smokeSCREEN involves a redirection of attention; the player’s response to the virtual stimuli, the theory goes, is reality.

“We are not competing with Grand Theft Auto or Fortnite,” Fiellin said. “What we’re trying to do is create a way to deliver [information] to kids in a space that they want to be in.”

smokeSCREEN imagines its player as a new kid in school; the object is to make friends while the player’s “risk” level fluctuates. Higher risk levels mean an inability to counteract peer pressure in battle royal-esque scenarios — the school is a hivemind of students who want nothing more than to see the player smoke. The game design was based on feedback from New Haven students; participants were asked to send pictures of their houses, their shoes, their hairstyles.

“A lot of times, kids know what the right decision is, and it’s just that they don’t live in a vacuum,” Fiellin said. “There’s a context that makes that decision-making challenging. We try to recreate that in our games.”

This context is not only visual but an element of gameplay: Players are free to experiment, making both the “wrong” and “right” choices in an environment that at least resembles their own, if on a cartoon plane. The game presents choices, and the choices feel real. If developers have done their job, the consequences do too.

—

“Nothing beats palpitating”

Of course, serious games are not necessarily good for learning, and game technology, though new, will not necessarily revolutionize medical research. If blended reality is a tool in an evolving toolbox, researchers must make decisions. The rusty screwdriver may be older, but sometimes it’s better, depending on what’s broken.

Travis McCann NUR ’20 has spent the past year testing tools. Ordinarily, educators at the School of Nursing create realistic patient simulations for nursing students by applying makeup to “high-fidelity Manikins,” mannequins with the ability to breathe, cry, even talk. Students are then tested on their ability to identify and address points of medical concern.

McCann, a Masters of Science in Nursing candidate and a member of Yale’s Blended Reality project, initially planned to use VR technology to do away with Manikins altogether, generating approximations of human injuries on virtual bodies. But there was something lacking, a tactile element he didn’t miss until it was gone.

“Nothing beats palpitating,” he said. “If you’re always just moving things around in the air, it’s not real. But when you put your hands on something, it’s real. … If you’re using VR, as soon as you step into a world that’s not the world you’re actually living in, there’s always that false sense of security that nothing can go wrong.”

McCann’s new goal is to implement augmented reality — game images layered atop a mobile device’s video capture — into the nursing curriculum. The Manikins, which lie eerily still in simulation rooms, mouths open, will not be forced to retire. With McCann’s technology, students will either use QR codes or the Microsoft Hololens to witness, for instance, a video of a tick stuck in the Manikin’s leg.

The Blended Reality project, a cross-disciplinary exploratory opportunity for Yale affiliates, originated in 2016 to support projects like McCann’s, in particular those whose outcomes or even methodology are unknown. The project relies on the fact that no developer can foretell VR’s future with any confidence. It’s largely untested in classrooms, research and treatment. Researchers might spend years on a project that functions but does little else or doesn’t function at all.

Justin Berry, critic at the Yale School of Art and principle investigator of the 2018-2019 Blended Reality project, believes this trial and error is an inevitable feature of untested technology. He contrasts it with drawing — students in his classes will claim they’ve only been drawing since high school, when really they’ve been drawing all their lives.

“Every time they picked up a pencil, they’re slowly having this content feedback mechanism,” he said. “When it comes to VR or AR or these immersive technologies, we don’t have that history of use.”

At this point, with significant time invested, learning is the only option.

“Say we make something and it’s perfect and it takes off on day one. Everyone uses it to save the world, congratulations,” he said. “I’d love to do that, but then you don’t actually learn much about why it works. You don’t really know much about it.”

—

Research and/or commerce

Investment is undeniably crucial. The slow crawl of serious games — including games used for research purposes — into both education and medicine generates gray-area questions of not only longevity but sustainability. Serious games, often funded by organizations such as the National Institutes of Health or private corporations, operate on a different financial wavelength than commercial games, which are intended to be mass-produced and sold.

While Shic attributes the spike of serious games to new, widespread knowledge about game technology, he believes how far such technology advances will depend on how deep NIH or private pockets stretch. With typical research funding from the NIH, he said, researchers are rarely motivated to complete projects; once completed, funding dries up.

Serious games thus exist at a unique crossroads. If Shic is right that serious game development is a scientific process, then on the one hand, researchers should drive development, as they understand the genre’s future possibilities from a scientific, and often a psychological, standpoint. On the other hand, game development depends on a transactional business model, which centers around commercial production.

For most researchers, psychology comes first. François, who has worked on exposure therapy prototypes, emphasized that blurring the “real” and the “make-believe” often requires an academic pedigree — in a game built to address claustrophobia in elevators, for instance, you don’t need precise realism. The number of virtual individuals in an elevator or the elevator’s suggested cleanliness might be all that’s needed to trigger a response.

“The games and experience interventions we create are rooted in what we already know,” Kim Hieftje said regarding smokeSCREEN VR’s development. Hieftje is the director of the play4REAL lab. “It’s not like we’re creating something new.”

Lebowitz believes YIKES might someday be used as a form of exposure therapy, and Fiellin hopes that smokeSCREEN will find its way into classrooms across the nation. Neither are interested in widespread commercial dissemination for a large profit margin. They have an idea with psychological underpinnings; established gaming companies have a means of mass-producing it. The technology exists, and most researchers believe it will expand, but how significantly depends on who blinks first, or whether both sides of development, research and game companies, can work to answer the same questions. If not, and perhaps regardless, the medium is one giant unknown. As to whether or not blended realities will replace researchers altogether, Shic turns a skeptical eye.

“My personal belief is that the human aspect of [psychology] is inescapable,” he said. “There’s nothing, no substitute, no technology or algorithm that is going to get us there because ultimately someone cares about you when you’re talking to a therapist. It’s going to be a hard sell to convince people that this algorithm cares about you.”

Randy Rode, director of Campus IT Partner Relationship and Development and of the Blended Reality project, believes blended reality is moving too quickly to halt altogether. The real question, then, is how significantly it will alter what we see as the boundaries of our own reality. Meanwhile, mothers and their children will continue to dance around spiders.

“It’s a good time to be skeptical, to be cautious, to ask a lot of questions,” Rode said. “But none of those are reasons not to do anything.”

—

Who’s updating whom?

I spent many hours playing smokeSCREEN from a computer in my childhood bedroom over winter break, a simulation built of both remembered childhood and game technology. The game is buoyant, even addictive; much of it revolves around conversations with potential new friends, each eventually classified as a “smoker” or a “nonsmoker.” There was something comforting about such binary distinctions within an opaque medium, such a singular and solvable problem — I was asked by my new friend Brian why I didn’t want to smoke if the characters in “Grease” did and chose to cordially reply, “When actors smoke or vape in movies, people get the impression that it’s cool.” Brian, ever-receptive, accepted this rote recitation of curriculum basics.

The trailer for smokeSCREEN VR demonstrates the same general philosophy, though in a VR space. The player walks through the high school hallway as their friends implore them to score a party invitation. It’s startling, perhaps, to think that kids may absorb health education via three-dimensional cartoons. Some things, though, remain familiar. In the trailer, a new friend turns to the player: “Hey, kid. You thirsty for some mango juice?” I couldn’t help but laugh, imagining a teenager’s reaction, imagining the adult who wrote the line.

But the kinks of the game will be worked out, and the format isn’t going away. If an algorithm can’t replace the study of health at large, then we aren’t adapting to game technologies. Game technologies — spiders and Manikins and cajoling cartoons — are adapting to us.