‘Smart aviary’ poised to break new ground in behavioral research
A collaboration that has brought together biologists, engineers, and physicists aims to study the reproductive behavior of birds using machine learning in a custom-built aviary at Pennovation Works.
A high-tech aviary at Penn’s Pennovation Works campus, equipped with arrays of cameras and microphones to record the behavior of 20 brown-headed cowbirds, is allowing biologists, physicists, and computer scientists to make advancements in our understanding of animal behavior, neuroscience, and machine learning.
Tucked away on the Pennovation Works campus in southwest Philadelphia, an 8’ by 8’ by 20’ wire cage represents what is likely the most high-tech aviary ever created. The enclosure is outfitted with eight computer-vision cameras, an array of 24 high-precision microphones, and 20 brown-headed cowbirds.
For biology professor Marc Schmidt, the aviary presents a more realistic setting than the lab to study the neural basis of reproductive behaviors in birds.
The goal of the aviary, originally envisioned by Marc Schmidt, a professor of biology in Penn’s School of Arts and Sciences, is to use the latest in machine learning technology to answer questions about animals’ social behavior that can be addressed in no other way.
“My student Ammon Perkes coined it the ‘smart aviary,’” Schmidt says, “because we can use this high-tech array of cameras and microphones to determine the position and identity of each individual bird and every sound that is being produced. But I like to call it the ‘cage of dreams.’ It’s sort of corny but I felt that, if we built it, the technology and the collaborators would come.”
Just as physicists and astronomers have pointed powerful telescopes at the sky, raking in vast quantities of data to later comb through, the Penn scientists working on the aviary are collecting data by the terabyte in their biological observatory, and then developing the tools and algorithms to parse it carefully to make new discoveries.
Schmidt has studied the neural basis of reproduction and song production in birds in his lab for close to two decades, but had long dreamed of setting up an aviary to examine these behaviors in a more naturalistic setting. In 2015, he reached out to Daniilidis and began discussing strategies to make this vision concrete.
I like to call it the ‘cage of dreams.’ It’s sort of corny but I felt that, if we built it, the technology and the collaborators would come.
Marc Schmidt, a professor of biology in Penn’s School of Arts and Sciences
Together with colleagues, they applied for and won a grant from the National Science Foundation’s Major Research Instrumentation Program to build the facility. Along the way, they brought into the fold Vijay Balasubramanian, a physicist in the School of Arts and Sciences who had pursued computational work in neuroscience at various levels, from neurons up to behavior.
“The thing I find very interesting about this project is it’s an attempt to completely analyze a developing social network,” Balasubramanian says. “As a physicist I would like to understand the formation of the collective behavior.”
Schmidt’s expertise in biology and experimental design, Daniilidis’s sophisticated computer vision systems, and Balasubramanian’s track record at analyzing complex systems made for a formidable interdisciplinary partnership.
A goal of the work is to use machine learning to discern specific poses and interactions of the birds within the aviary. The challenge for computer scientists is to use two-dimensional information from the cameras to infer the three-dimensional positions of the birds.
Dream team in place, the next step was construction. That effort was led by postdoctoral researcher Bernd Pfrommer, formerly of Daniilidis’s group, and doctoral student Perkes from Schmidt’s lab, who went from initial sketches in the proposal to a fully framed enclosure by 2017. The process required navigating a variety of challenges, from applying for permits to collect cowbirds from the wild, to figuring out how to arrange the cameras and microphones, to establishing a server capable of relaying large quantities of data from the Pennovation Works campus to the main Penn campus.
Part of the challenge was also communicating across disciplinary boundaries.
“There’s a different language between biologists and engineers,” says Perkes. “The engineers may see what they can do and not think about the fact that the birds are going to want to perch on the cameras—or poop on the cameras. But we’ve been learning to talk to one another so everyone is on the same page.”
The group began recording in the aviary during this past spring’s breeding season. With 10 female and 10 male cowbirds in the aviary, the cameras and microphones recorded 10 hours a day, capturing every wing stroke, head bow, tussle, song, and call—close to a terabyte of data a day (after 60-fold compression) for 100 days.
Researchers hope to use the data they collect to decipher the intricate set of interactions that go into a successful pairing between birds. “It’s a very complex collective of interactions,” says physicist Vijay Balasubramanian. “It’s not just ‘A talks to B.’ It’s ‘A talks to B and C’s opinion of A talking to B matters.’ It’s very complicated to figure out.”
Cowbirds are a gregarious species that dwell in groups, but also form breeding pairs each season. One aim of the work is to see how different interactions between the birds give rise to these stable bonds.
“It’s a very complex collective of interactions,” says Balasubramanian. “It’s not just ‘A talks to B.’ It’s ‘A talks to B and C’s opinion of A talking to B matters.’ It’s very complicated to figure out.”
Each bird in the aviary can be distinguished by colored leg bands, but Marc Badger, a postdoctoral researcher in Daniilidis’s group, is working to craft algorithms capable of discerning different poses of the birds based on their silhouettes. Females, for example, go into what is known as a copulatory response, a kind of submissive posture, to indicate that they are receptive of a male’s advances. The engineers’ task is to employ machine learning to distinguish these types of subtle movements from others.
“It’s much easier to transfer results from joint positions in humans, or even cheetahs or monkeys, animals that have clear articulation in their joints,” says Daniilidis. “In a bird it’s very difficult to click on a point and say, ‘This is the joint of articulation,’ because a lot of articulation happens underneath the wings, for example.”
As humans we’re limited to only observing one thing at a time. This presents a tool we can use to observe behaviors that have previously been invisible.
doctoral student Ammon Perkes
This task is further complicated by the difficulty of picking out bird silhouettes amid the shifting shadows that cross over the enclosure during the course of a day. And a recycling center and trash compactor located near the aviary complicate the job of isolating the vocalizations of the birds in the study from background noise.
“That is really an open challenge, we’re still trying to solve it,” Daniilidis says.
Undergraduate students are assisting in the effort to overcome these obstacles, “training” the computer vision system to identify birds by manually clicking on their locations in the thousands of hours of video that have been collected so far.
Within the facility, Perkes, Badger, and undergrads work daily to keep the cowbirds fed and watered and to troubleshoot as needed. When the birds began perching on the cameras, causing the image to wobble, they installed spikes to deter them. And when the yogurt-covered almonds the researchers were using as imitation eggs began to melt in the sun’s heat (cowbirds are parasites and typically lay eggs in the nests of other birds), they swapped in specially ordered replica eggs made of wood.
Doctoral student Ammon Perkes places faux eggs in a nest to tempt the parasitic cowbirds to lay their own eggs there. At the conclusion of the first season of recording, Perkes and the rest of the team have seen the birds pair off, lay eggs, and carry out other natural behaviors within the aviary.
With the first season of recording complete, the researchers are pleased with the progress.
“It’s been cool to see the bird behaviors [that we might see in the wild] in this aviary that we’ve built from the ground up,” says Perkes.
All of the team members have grand ambitions for the future. Schmidt, for instance, would like to expand the experimental work on brain circuits associated with singing and reproductive behavior that he’s conducted in the lab to the aviary.
“One of the rationales for this aviary was to do neurobiology,” Schmidt says. “There is a brain circuit that, in males, is associated with singing. We found out a while ago that if you make targeted lesions to that circuit in females, they will lose the selectivity of their responses, as if this circuit is somehow involved in one or more aspects of female mate choice. It would be fascinating to put a bird with that lesion in the aviary to see what happens to the social dynamics in the group.”
In a related line of research, Schmidt is embarking on a collaboration with biologists at Swarthmore College to investigate the role of hormones on brain circuits related to reproduction, with an aim of someday remotely recording information about brain activity and perhaps hormone levels in birds interacting socially in the aviary.
Together, the Penn group is also exploring additional applications for this kind of observatory, so the setup could be used by other research groups pursuing different scientific questions on a variety of species.
“I hope that this aviary can help push the field of behavior in a more analytical direction,” says Perkes. “As humans we’re limited to only observing one thing at a time. This presents a tool we can use to observe behaviors that have previously been invisible.”
In future research, Schmidt and colleagues would like to track the inner biological workings of birds, such as fluctuations in hormones or brain activity, as they interact with one another in the aviary.
Marc Schmidt is a professor in the School of Arts and Sciences’ Department of Biology at the University of Pennsylvania.
Kostas Daniilidis is the Ruth Yalom Stone Professor in the Department of Computer and Information Science in Penn’s School of Engineering and Applied Science.
Vijay Balasubramanian is the Cathy and Marc Lasry Professor of Physics in the Department of Physics in Penn’s School of Arts and Sciences.
Ammon Perkes is a Ph.D. candidate in the Schmidt lab in the Department of Biology in Penn’s School of Arts and Sciences.
Marc Badger is a postdoctoral researcher in the Department of Computer and Information Science in Penn’s School of Engineering and Applied Science working with Daniilidis and Schmidt.
Bernd Pfrommer is a former postdoctoral researcher in Penn’s School of Engineering and Applied Science.
Materials in the Annenberg School for Communication Library Archives include thousands of TV scripts, the first issue of TV Guide, and interviews about the early days of HBO—which help to chronicle TV’s 100-year story.
Centering joy in AI development and implementation
PIK Professor Desmond Upton Patton—of Annenberg and SP2—and collaborators introduce a joy-informed framework designed to initiate conversations among engineers, designers, and researchers.
Winter Storm Fern brought icy and snowy conditions to the Northeast and other parts of the country over the weekend. Penn Today asks physicist Robert Carpick about the unique properties of ice, the science of curling, and how close we are to ‘nonslip’ ice.
Organizations like Penn’s Netter Center for Community Partnerships foster collaborations between Penn and public schools in the West Philadelphia community.
Penn receives national distinction for community engagement
The recognition by the American Council on Education and Carnegie Foundation for the Advancement of Teaching acknowledges Penn’s long-standing commitment to community-engaged scholarship and partnerships in West Philadelphia and beyond.
