Students use machine learning to track and protect whale populations

View large image

As whales face harm from ship strikes, fishing net entanglements, and redistribution of prey due to changes in ocean temperature, it’s increasingly important to track their locations and populations across the globe. To help accelerate these efforts, second-year Chinmay Govind and third-year Nihar Ballamudi dedicated their summer to a Penn Undergraduate Research Mentoring Program (PURM) project that combines mathematics, signal processing, animal behavior, and machine learning.

Their goal: Leverage whale sound data and artificial intelligence to map the locations of whales and determine how many live in any given target area.

For this work, Govind and Ballamudi have used National Oceanic and Atmospheric Administration (NOAA) data from sound receivers north of Cape Cod Bay, though their research applies to any location.

Results from this endeavor could help the duo obtain “better data on how many whales are in an area or the distribution of whales in an area, which can inform policymakers and environmental groups on policies involving whales,” says Govind, a double major in artificial intelligence and computer engineering in the School of Engineering and Applied Science and originally from Mechanicsburg, Pennsylvania. “The findings of our research can extend not just to whales, but [also] other sea animals.”

PURM, offered by the Center for Undergraduate Research & Fellowships, immerses students finishing their first or second year at Penn in a 10-week summer research experience under the expert guidance of a faculty mentor.

Ballamudi and Govind are mentored by John Spiesberger, a visiting scholar in the Department of Earth & Environmental Science, along with his son, Ari Spiesberger, a recent Penn graduate with expertise in machine learning models. Both students took interest in the whale monitoring project—which is sponsored by Joseph Kroll, a professor in the Department of Physics and Astronomy and John Spiesberger’s colleague of five decades—because of its multidisciplinary, problem-solving nature, as well as the tangible impact it could have for whale conservation efforts worldwide.

“Math research isn’t really used that often outside of, you know, just math,” says Ballamudi, a mathematics major in the College of Arts & Sciences and computer science minor from Madison, Wisconsin. “It’s really cool for me to be able to work on a project that [uses math to] help influence what policy will look like if we can census whales.”

Leveraging sound to locate whales

Each student led a portion of the PURM project: Govind focused on locating whales, and Ballamudi worked on censusing them. In this context, locating entails tracking and counting individual whales. Censusing, on the other hand, involves approximating the size and distribution of whale populations to more effectively monitor their movements.

To locate whales, Govind leveraged acoustic data from NOAA receivers—essentially underwater microphones—to estimate the origin points of whale calls. Each receiver detects the sound waves from a unique whale call at different times. Govind feeds the recorded audio data into a machine learning model to estimate “time difference of arrival,” which is then used to calculate the whale’s coordinates—similar to how mobile phones derive their locations using GPS.

View large image

“Time difference isolates sound to a specific curve,” Govind explains. “If you have more receivers—we use five—then you have enough data to pinpoint the whale’s position or generate a confidence interval for where the whale could be.”

There are several barriers, however, to finding the time difference of arrival for a whale call. Ocean noise can distort receiver data, making it difficult to discern where a whale call starts and stops—or whether several whales are producing sounds within close proximity. There’s also “multipath”—excessive sound that echoes off the ocean floor and surface—which can interfere with receiver data.

“This data can be very messy. That’s why we’ve decided to apply machine learning,” Govind says.

Using AI to optimize and refine acoustic data, Govind has been able to record the origin points of whale calls with a “median error of 20 milliseconds.” This small margin of uncertainty, he says, is more than sufficient for estimating whale locations.

Building a census of whale whereabouts

Concurrently, Ballamudi used machine learning models and NOAA sound data to simulate sea environments and census whale populations. This AI-driven approach can be more effective than relying on data from physical receivers given the obstacles posed by ocean noise and multipath.

“We’ve sampled real ocean noise and generated signals according to literature regarding what whale signals will usually look like,” Ballamudi says. “Using that information, we’re able to generate as much data as we want.”

This strategy allows Govind and Ballamudi to innovate as they learn about individual and collective whale behavior.

During the PURM project, Ballamudi has accurately predicted the number and distribution of whales between 90-95% of the time—a significant feat given the challenges involved with censusing whales.

Deep dives in mentorship

Spiesberger has focused much of his guidance on showing Govind and Ballamudi “the wonders of what a realistic simulation looks like in this case, and what kind of variation we should put in the simulation” to mirror ocean settings and produce meaningful data. He also has coached the duo to enhance their science communication skills, especially given the environmental policy implications of their work.

“We practice talking to people in a few minutes about what they do without using jargon,” Spiesberger says.

Ari Spiesberger paid it forward as a Penn alum, training Govind and Ballamudi to use AI models, build realistic simulations, and prime the software to predict unknown variables.

The AI models used for this PURM project also continually optimize and improve their precision, showing promise for future steps the students could take with this research.

View large image

“It would be very nice if we could get a model to recognize multiple sources at the same time and be able to look at all of them in one shot,” Govind says, noting how current limitations allow them to locate only one whale at a time.

Once the pair can record the exact number of whales in a target range, Ballamudi says, they could leverage that data to retroactively pinpoint the precise location of each whale.

“We want to see if this approach will work no matter what—not just in our well-controlled software, but also in a world that has way more confounding variables than our software could ever account for,” Ballamudi says.

The tangible power of PURM

Toward the end of their PURM experience, Spiesberger invited Govind and Ballamudi to present their results to U.S. Navy sponsors; highlight the implications for policymakers working to protect whales; and share opportunities for expanding upon this research.

“It’s important for the Navy to know where whale sounds are coming from, and this PURM project will help solve that problem,” says Spiesberger, who aims to secure grants for Govind and Ballamudi so they can continue this impactful work long after summer concludes. “I hope to get funding for them to support their research in the future.”