Tucked away on the Pennovation Works campus in southwest Philadelphia, an 8’ by 8’ by 20’ wire cage represents what is likely the most high-tech aviary ever created. The enclosure is outfitted with eight computer-vision cameras, an array of 24 high-precision microphones, and 20 brown-headed cowbirds.
The goal of the aviary, originally envisioned by Marc Schmidt, a professor of biology in Penn’s School of Arts and Sciences, is to use the latest in machine learning technology to answer questions about animals’ social behavior that can be addressed in no other way.
“My student Ammon Perkes coined it the ‘smart aviary,’” Schmidt says, “because we can use this high-tech array of cameras and microphones to determine the position and identity of each individual bird and every sound that is being produced. But I like to call it the ‘cage of dreams.’ It’s sort of corny but I felt that, if we built it, the technology and the collaborators would come.”
Just as physicists and astronomers have pointed powerful telescopes at the sky, raking in vast quantities of data to later comb through, the Penn scientists working on the aviary are collecting data by the terabyte in their biological observatory, and then developing the tools and algorithms to parse it carefully to make new discoveries.
“For us it’s very exciting science,” says Kostas Daniilidis, the Ruth Yalom Stone Professor in the School of Engineering and Applied Science’s Department of Computer and Information Science. “It gives us the chance to work on translating two-dimensional data to three-dimensional data, and it also gives us a chance to use AI tools to recognize complex poses in the birds.”
Schmidt has studied the neural basis of reproduction and song production in birds in his lab for close to two decades, but had long dreamed of setting up an aviary to examine these behaviors in a more naturalistic setting. In 2015, he reached out to Daniilidis and began discussing strategies to make this vision concrete.
Together with colleagues, they applied for and won a grant from the National Science Foundation’s Major Research Instrumentation Program to build the facility. Along the way, they brought into the fold Vijay Balasubramanian, a physicist in the School of Arts and Sciences who had pursued computational work in neuroscience at various levels, from neurons up to behavior.
“The thing I find very interesting about this project is it’s an attempt to completely analyze a developing social network,” Balasubramanian says. “As a physicist I would like to understand the formation of the collective behavior.”
Schmidt’s expertise in biology and experimental design, Daniilidis’s sophisticated computer vision systems, and Balasubramanian’s track record at analyzing complex systems made for a formidable interdisciplinary partnership.
Dream team in place, the next step was construction. That effort was led by postdoctoral researcher Bernd Pfrommer, formerly of Daniilidis’s group, and doctoral student Perkes from Schmidt’s lab, who went from initial sketches in the proposal to a fully framed enclosure by 2017. The process required navigating a variety of challenges, from applying for permits to collect cowbirds from the wild, to figuring out how to arrange the cameras and microphones, to establishing a server capable of relaying large quantities of data from the Pennovation Works campus to the main Penn campus.
Part of the challenge was also communicating across disciplinary boundaries.
“There’s a different language between biologists and engineers,” says Perkes. “The engineers may see what they can do and not think about the fact that the birds are going to want to perch on the cameras—or poop on the cameras. But we’ve been learning to talk to one another so everyone is on the same page.”
The group began recording in the aviary during this past spring’s breeding season. With 10 female and 10 male cowbirds in the aviary, the cameras and microphones recorded 10 hours a day, capturing every wing stroke, head bow, tussle, song, and call—close to a terabyte of data a day (after 60-fold compression) for 100 days.
Cowbirds are a gregarious species that dwell in groups, but also form breeding pairs each season. One aim of the work is to see how different interactions between the birds give rise to these stable bonds.
“It’s a very complex collective of interactions,” says Balasubramanian. “It’s not just ‘A talks to B.’ It’s ‘A talks to B and C’s opinion of A talking to B matters.’ It’s very complicated to figure out.”
Each bird in the aviary can be distinguished by colored leg bands, but Marc Badger, a postdoctoral researcher in Daniilidis’s group, is working to craft algorithms capable of discerning different poses of the birds based on their silhouettes. Females, for example, go into what is known as a copulatory response, a kind of submissive posture, to indicate that they are receptive of a male’s advances. The engineers’ task is to employ machine learning to distinguish these types of subtle movements from others.
“It’s much easier to transfer results from joint positions in humans, or even cheetahs or monkeys, animals that have clear articulation in their joints,” says Daniilidis. “In a bird it’s very difficult to click on a point and say, ‘This is the joint of articulation,’ because a lot of articulation happens underneath the wings, for example.”
This task is further complicated by the difficulty of picking out bird silhouettes amid the shifting shadows that cross over the enclosure during the course of a day. And a recycling center and trash compactor located near the aviary complicate the job of isolating the vocalizations of the birds in the study from background noise.
“That is really an open challenge, we’re still trying to solve it,” Daniilidis says.
Undergraduate students are assisting in the effort to overcome these obstacles, “training” the computer vision system to identify birds by manually clicking on their locations in the thousands of hours of video that have been collected so far.
Within the facility, Perkes, Badger, and undergrads work daily to keep the cowbirds fed and watered and to troubleshoot as needed. When the birds began perching on the cameras, causing the image to wobble, they installed spikes to deter them. And when the yogurt-covered almonds the researchers were using as imitation eggs began to melt in the sun’s heat (cowbirds are parasites and typically lay eggs in the nests of other birds), they swapped in specially ordered replica eggs made of wood.
With the first season of recording complete, the researchers are pleased with the progress.
“It’s been cool to see the bird behaviors [that we might see in the wild] in this aviary that we’ve built from the ground up,” says Perkes.
All of the team members have grand ambitions for the future. Schmidt, for instance, would like to expand the experimental work on brain circuits associated with singing and reproductive behavior that he’s conducted in the lab to the aviary.
“One of the rationales for this aviary was to do neurobiology,” Schmidt says. “There is a brain circuit that, in males, is associated with singing. We found out a while ago that if you make targeted lesions to that circuit in females, they will lose the selectivity of their responses, as if this circuit is somehow involved in one or more aspects of female mate choice. It would be fascinating to put a bird with that lesion in the aviary to see what happens to the social dynamics in the group.”
In a related line of research, Schmidt is embarking on a collaboration with biologists at Swarthmore College to investigate the role of hormones on brain circuits related to reproduction, with an aim of someday remotely recording information about brain activity and perhaps hormone levels in birds interacting socially in the aviary.
Together, the Penn group is also exploring additional applications for this kind of observatory, so the setup could be used by other research groups pursuing different scientific questions on a variety of species.
“I hope that this aviary can help push the field of behavior in a more analytical direction,” says Perkes. “As humans we’re limited to only observing one thing at a time. This presents a tool we can use to observe behaviors that have previously been invisible.”
Marc Schmidt is a professor in the School of Arts and Sciences’ Department of Biology at the University of Pennsylvania.
Kostas Daniilidis is the Ruth Yalom Stone Professor in the Department of Computer and Information Science in Penn’s School of Engineering and Applied Science.
Vijay Balasubramanian is the Cathy and Marc Lasry Professor of Physics in the Department of Physics in Penn’s School of Arts and Sciences.
Ammon Perkes is a Ph.D. candidate in the Schmidt lab in the Department of Biology in Penn’s School of Arts and Sciences.
Marc Badger is a postdoctoral researcher in the Department of Computer and Information Science in Penn’s School of Engineering and Applied Science working with Daniilidis and Schmidt.
Bernd Pfrommer is a former postdoctoral researcher in Penn’s School of Engineering and Applied Science.
