When Charles Wang, a student at the University of Pennsylvania, was growing up, he wanted to be a video-game designer. As he headed to college, he came to believe he would have to trade in this goal for a more common profession.

“It's one of those things you just say and you have no idea what it implies,” he says. “When you go into high school and then start college, you kind of give up on the idea because it sounds unrealistic.”

As a first-year undergraduate at Penn, Wang declared his major in computer engineering. But he couldn’t kick his childhood dream of working in the game industry. In his sophomore year, he transferred into the Digital Media Design program and eventually sub-matriculated into the Computer Graphics and Game Technology master’s program.

One course Wang took through this program is the Game Design Practicum, a course that introduces students to the latest tools and engines used in game development, including virtual and augmented reality, to allow them to create original content.

“At Penn,” Wang says, “we have such great resources for programs like this, and it's just so cool that these really outlandish, almost childish dreams can actually become serious.”

In the Game Design Practicum, students learn to work with leading industry game engines such as Unity3D and Unreal Engine 4, hardware such as the Oculus Rift, the HTC Vive and the Microsoft Hololens, and they use combinations of C++ and C# to code their own environments.

“I think Penn is pretty unique in being an Ivy League university that not only focuses a lot on the theory of the things but also provides practicum-based training,” says Sacha Best, a recent Penn graduate who was also a teaching assistant for the Practicum class which is taught by Stephen Lane, a professor of practice in the Department of Computer and Information Science in the School of Engineering and Applied Science. “Having a class where you can kind of take those things that you've learned over the past four years, apply them and see something truly visual and compelling that can transport you to a different world is a really rare opportunity and something that I'm very grateful to be a part of.”​​​​​​​The DMD program, an interdisciplinary undergraduate program in computer science that combines art and engineering, was founded in 1999 by Norman Badler, the Rachleff Professor of Computer and Information Science at Penn. In this program, students take courses in computer science, fine arts and communications.

In 2004, Lane, who had experience running a computer graphics company in Princeton, N.J., that had developed virtual reality games, 3D animation and immersive training simulations for companies such as Disney, Hasbro and the US Army, put together the CGGT master’s program. He says he saw a need for people with interdisciplinary skills in graphics and game development, and, at the time, there were no other such master’s programs available.

“The way I describe it,” he says, “is that there's a creative spectrum. On the one side you have artists and animators, and on the other you might have hardcore programmers, software engineers and systems people. And so our program is focused on the center of the creative spectrum where it's not about just working with technology but trying to put it in a form that creative people can use. If the technology is just raw and it's not user-friendly, then nobody's going to do anything with it.”

Students in the DMD and CGGT programs, Lane says, are well versed in the math and science behind the technology as well as the creative process and possible applications for it. He says that, after the program, students often go on to work in the animation industry, game industry or for companies such as Microsoft, Google or Facebook.

“We're just trying to give students a solid grounding in the basic principles of graphics and animation,” he says, “as well as enough hands-on experience with game development and using the latest virtual reality devices so that, when they go out to look for a job or work in industry, they can show some impressive demos and talk from a position of strength. They not only can create the tools but also figure out how to effectively use them.”

To Lane, the most exciting part of these programs is seeing students go through a sequence of courses when in the beginning they may not known anything about graphics and animation, and then a year or two later they’re creating interesting and novel content with real commercial potential.

“They’ve gone through that whole process of mastering the technology but also understanding the creative process and what makes something, at least from a game or VR perspective, interesting or fun.”Last spring, students in the Game Design Practicum created a variety of games using VR and AR technologies. Wang’s final project, which he worked on with CGGT student Rachel Hwang and Mohamad Moneimne, a computer science major who is also sub-matriculating into the CGGT program, was a VR game called Angry Beard, in which an angry wizard fends off opponents coming to steal his power. To cast different spells, players have to move two hand-held controllers in specific patterns.[vimeo]196372438[/vimeo]

“I really like the freedom that we get to explore,” Moneimne says. “This is one of the capstone courses of the graphics programs. You get dropped into these game engines that have so much power, and it gives you the opportunity to take everything you've learned and make something with it.”

Daniel Daley-Montgomery, Katie Wu and Nick Newberg, all DMD students, worked together to create both a virtual reality game and an augmented reality game. Their virtual reality game Ratchet was inspired by a scene in Ratatouille where a chef destroys his own kitchen running around trying to chop a mouse. But in their game, the tiny mouse running around the kitchen is actually a person, and the chef is actually a giant mouse.
[vimeo]192724434[/vimeo]

“The whole draw I originally had to computer graphics and programming in general was the power to create new worlds,” Daley-Montgomery says. “I started out making 3-D art and eventually interactive programs. Bu,t once you convert to virtual reality, you’re creating entirely new immersive realities. It multiplies the entire feeling.”

Daley-Montgomery, Wu and Newberg also created an augmented reality game called Ourside, which enables players to draw art on top of their real world surroundings. Ourside, Wu says, is like “an alternate dimension where players can doodle and share art and creation together.”
[vimeo]196290087[/vimeo]

“We wanted to create this world that was persistent,” Newberg says, “so when you turn the headset off and then you come back into the room a couple days later, your creations are still there. It’s leaving a mark in a way, kind of like graffiti, and the idea is that maybe in the future we could get into this world with our friends and actually collaborate on drawings and sculptures together.”

Ellen Duong, Gary Li and Grace Xu, students in the DMD program, created an augmented reality fishing game using the Hololens and an Android phone for the controller. The students designed five different types of fish, including a traditional koi fish, a pocket whale and a “credit card phish.”
​​​​​​​[vimeo]196215095[/vimeo]

“In the real world environment,” Duong says, “it's really hard to go fishing because you have to go get all your gear and check the weather and everything. But with AR fishing you get to bring that experience to your home or work or anywhere.”

Xu, who is also sub-matriculating into the CGGT program, says it’s “all the fun of fishing without the mess.”

When Duong started this program, she says she knew she liked computer science and art and that she wanted to use them to create new experiences, but she wasn’t sure how they really aligned.

“I think this program really gave me the tool set necessary to build those immersive environments that I've wanted to create and be a part of,” she says.