Insights into how computers learn, the current challenges of artificial intelligence research, and what the future holds for how machines might shape society in the future.
A robot from Penn’s Rehabilitation Robotics Lab. Whether the machines are commercially available, like this Baxter model, or made in-house here at Penn, fundamental research on artificial intelligence is key to creating machines that can work with, and for, people more effectively.
The phrase “artificial intelligence” might conjure robotic uprisings led by malevolent, self-aware androids. But in reality, computers are too busy offering movie recommendations, studying famous works of art, and creating fake faces to bother taking over the world.
During the past few years, AI has become an integral part of modern life, shaping everything from online shopping habits to disease diagnosis. Yet despite the field’s explosive growth, there are still many misconceptions about what, exactly, AI is, and how computers and machines might shape future society.
Part of this misconception stems from the phrase “artificial intelligence” itself. “True” AI, or artificial general intelligence, refers to a machine that has the ability to learn and understand in the same way that humans do. In most instances and applications, however, AI actually refers to machine learning, computer programs that are trained to identify patterns using large datasets.
Read more from this series on artificial intelligence
“For many decades, machine learning was viewed as an important subfield of AI. One of the reasons they are becoming synonymous, both in the technical communities and in the general population, is because as more data has become available, and machine learning methods have become more powerful, the most competitive way to get to some AI goal is through machine learning,” says Michael Kearns, founding director of the Warren Center for Network and Data Sciences.
If AI isn’t an intelligent machine per se, what, exactly, does AI research look like, and is there a limit to how “intelligent” machines can become? By clarifying what AI is and delving into research happening at Penn that impacts how computers see, understand, and interact with the world, one can better see how progress in computer science will shape the future of AI and the ever-changing relationship between humans and technology.
Intelligent machines?
All programs are made of algorithms, “recipes” that tell the computer how to complete a task. Machine learning programs are unique: Instead of detailed step-by-step instructions, algorithms are “trained” on large datasets, such as 100,000 pictures of cats. Machine learning programs then “learn” which features of the image make up a cat, like pointed ears or orange-colored fur. The program can use what it learned to decide whether a new image contains a cat.
Computers excel at these pattern-recognition tasks, with machine learning programs able to beat human experts at games like chess or the Chinese board game GO, because they can search an enormous number of possible solutions. According to computer scientist Shivani Agarwal, “We aren’t designed to look at 1,000 examples of 10,000 dimensional vectors and figure out patterns, but computers are terrific at this."
For machine learning programs to work well, computers need a lot of data, and part of what’s made recent AI advances possible is the Internet. With millions of Facebook likes, Flickr photos, Amazon purchases, and Netflix movie choices, computers have a huge pool of data from which to learn. Coupled with simultaneous technological improvements in computing power, machines can analyze massive datasets faster than ever before.
But while computers are good at finding cats in photos and playing chess, pattern recognition isn’t “true” intelligence—the ability to absorb new information and make generalizations. As Agarwal explains, “These are not what we would call ‘cognitive abilities.' It doesn’t mean that the computer is able to reason.”
“Most of the successes of machine learning have been on specific goals that nobody would call general purpose intelligence,” says Kearns. “You can’t expect a computer program that plays a great game of chess to be able to read today’s news and speculate on what it means for the economy.”
See the world
The human brain devotes more neurons to sight than the other four senses combined, providing the “computing power” needed to see the world.
Examples of how perspectives change whether a person, or a robot, is doing a task or watching someone else do that same task. (Image: Jianbo Shi)
Computer vision researchers study ways to help computers “see”—and an accurate visual representation of the environment is a crucial first step for AI, whether it’s facial recognition software or self-driving cars.
Computer vision programs are able to learn from massive datasets of human-curated images and videos, but one of the hurdles faced by researchers is getting computers to see what’s not actually there. Computer vision researcher Jianbo Shi gives the example of the Kanizsa triangle, an optical illusion in which a triangle can be clearly perceived by the human eye, even though there aren’t explicit lines outlining the shape. Human brains “hallucinate” the missing parts of the triangle, but computers cannot see it at all.
One of the ways that Shi is helping computers see better is using first-person GoPro videos, which provide computers a more accurate and fuller perspective on a human activity so they can make more accurate predictions and decisions. “It’s one thing to look at somebody do something,” Shi says. “It’s another thing to experience it from your own point of view.” These egocentric programs are able to predict a player’s movements or find a group’s collective point of attention.
Using first-person cameras, Shi and his research group can use images captured during a one-on-one basketball game and translate the visual information into a projected path (Video credit: Gedas Bertasius).
Shi hopes that this research can not only lead to improvements in AI platforms but could also benefit people, like making it easier to learn a musical instrument or play a sport . “We hope we can use technology to teach humans skills that would otherwise take too long to learn. They say it takes 10,000 hours to perfect some skill, but can we shorten this time if we use technology?” Shi says.
Speak the language
Humans communicate using ambiguous languages. For example, a “break” can be an abrupt interruption, a bit of luck, an escape from jail, or one of 13 other meanings.
Research in natural language processing works to clarify ambiguous words and phrases to help humans communicate with computers. It’s an area of study that’s fundamental to AI, especially as voice-recognition platforms like Alexa and Siri become more popular. “Being able to make inferences about language is important, and if we want agent-human interaction, that’s absolutely a must,” says computational linguist Ani Nenkova.
Nenkova’s research group works at the interface of computational linguistics and AI, and is focused on developing ways to help computers better analyze text for less apparent metrics such as quality and style. (Image: Penn Engineering)
Humans learn languages by exposure, either from hearing others speak or from rigorous study. Computers can gain exposure to language using digitized text or voice recording datasets, but still need help from humans to understand the exact meaning of what is being said. As with the Kanizsa triangle example, computers often misinterpret or struggle to understand things that are left unsaid which might be clear to the human listener.
In Nenkova’s research on what makes “great” writing and literature search automation, she trains programs on word representation datasets curated by humans that tell the computer what words and phrases mean in a specific context. The long-term goal is to develop new algorithms that can analyze and understand new text without a human “translator,” but that’s still many years in the future.
One complex problem, faced by both natural language processing researchers and broader AI research as a whole, is shared knowledge. For example, if someone asks her friend, “How was the concert last night?” but the friend didn’t go to a concert, clarifying the misunderstanding is straightforward. Computers don’t realize that they lack some information or common ground and would instead give a faulty answer in response.
“Shared knowledge is [the idea of] how can the machine figure out that a person is expecting them to know something that they don’t know—having a sense of the knowledge they have versus the knowledge they need,” says Nenkova. Shared knowledge also relates to understanding a phrase’s deeper meaning, and Nenkova hopes that helping computers understand language can improve their awareness of what they know and don’t know.
By addressing this and other challenges, Nenkova also anticipates that natural language processing research could improve interpersonal communication. “There’s so much that we assume as common ground, and sometimes there is no common ground. If we try to address self-awareness, it may help people as well,” says Nenkova.
Perceive and move
“We look at robotics as embodied AI,” says roboticist Kostas Daniilidis. “Something which has sensors to receive [information] and motors to interact with the world.”
In the broad realm of AI research, roboticists face the additional challenge of interacting and reacting with chaotic, real-world environments. “Google uses AI to recommend things, and if they are wrong one out of five times, it’s annoying. For robotics, it has to work as well as a bar code in the supermarket,” says Daniilidis.
We aren’t designed to look at 1,000 examples of 10,000 dimensional vectors and figure out patterns, but computers are terrific at this.
Computer scientist Shivani Agarwal
Researchers start by giving robots lots of data and simulated experiences, but simply having more data isn’t enough for a robot to accurately translate a task’s complex physics into an appropriate action. To give robots more real-world experience, Daniilidis and Shi collaborate with Vijay Balasubramanian on ways to create “curious” robots. “Instead of methods where you teach [a computer] ‘Here’s a car, here’s a person,’ we are trying to think about how children learn,” says Shi.
The challenge is that robots can be programmed to look for patterns, but they won’t explore, like a child does, without a specific task. As a first step toward this goal, researchers have programmed a robotic arm to move around randomly and “explore” a box of assorted items like toys, clothes, and sporting goods.
With this data as a starting point for developing their algorithms, Penn researchers will then assign new robots a specific task, such as moving from Point A to Point B in a complicated multilevel setting, but give it days to finish a task that should only take a couple of hours. Their goal is to create a “curious” robot that uses the additional time it’s been given to explore a new environment so that it can complete future tasks more efficiently.
As part of the curiosity project, robotic systems are programmed to move around and interact with random objects. This data will then be a starting point for developing more advanced algorithms and physical systems that can interact with unknown environments and objects.
“The way we understand [curiosity] is that when we are performing a task, we have more time than needed,” Daniilidis says. “If you start your homework at 10 p.m. and you have to finish it by midnight, you’re not going to exhibit any curious behavior. But if you have one week, you [might].”
The future of AI
While the chances of creating a truly intelligent, self-aware robot are low, AI is still a powerful tool to be wielded wisely and understood clearly. It’s a completely new type of technology, one that’s deeply connected to the human experience, including all of society’s biases and social constraints, because computers rely on humans to “learn” about the world.
It’s why researchers like Agarwal emphasize the importance of establishing core principles for AI that clearly define success and failure in AI platforms, and that indicate when algorithms work well and when their use might be harmful.
“We want to improve quality of life by doing things that were not possible earlier, but we need to have principles,” emphasizes Agarwal. “Once we have a clear understanding of the principles, we can design the algorithms accordingly and implement them with computers. Computers are really at our beck and call. The challenge is for us as humans to come together and decide what is acceptable for us to ask of them.”
In Senegal, the ambitious Dakar Greenbelt project seeks to create an extensive network of ecological infrastructure in and around the city to sustainably address environmental concerns and enhance urban life. With support from David Gouverneur and Ellen Neises, Ph.D. candidate Rob Levinthal in the Weitzman School of Design led two courses that included a field trip to Dakar, that culminated in students presenting their visions for parts of the Greenbelt.
From a desert to an oasis: Penn engages in ambitious greening effort in the Sahel
Students from the Weitzman School of Design journeyed to Senegal to help with a massive ecological and infrastructural greening effort as part of their coursework. The Dakar Greenbelt aims to combat desertification and promote sustainable urban growth.
As part of an undergraduate course, Penn faculty and students curated an Arthur Ross Gallery exhibition of works from the Neumann family’s extensive collection of modern and contemporary art.
The University’s nexus for technology transfer supports researchers in their innovative efforts, from CAR T to mRNA advancements that have dramatically reshaped the world.
