Penn Engineering’s multimodal medical dataset, Observer, links video, audio, and transcripts to clinical data and electronic health records.
2 min. read
Researchers at the School of Engineering and Applied Science have launched Observer, the first multimodal medical dataset to capture anonymized, real-time interactions between patients and clinicians, allowing outsiders to peer inside primary care clinics.
Kevin Johnson, left, demonstrating the recording process with Karen O’Connor, right.
(Image: Sylvia Zhang)
Until now, the data available to health care researchers has been limited to traces left behind after a visit: qualitative information like clinician notes and quantitative measurements like patient vital signs. None of these sources capture subtleties like body language and vocal tone, or the environmental factors, including computer use, that affect how providers and patients engage with one another.
“So much of what shapes medical visits and their outcomes has been invisible to researchers,” says Kevin B. Johnson, David L. Cohen University Professor and the lead author of a new paper describing Observer in the Journal of the American Medical Informatics Association. “Thanks to technology that anonymizes our recordings, enabling HIPAA compliance, Observer lets us watch care unfold. That kind of evidence isn’t just the foundation for improving clinical practice, it’s crucial for developing responsible AI tools to augment care.”
The researchers have already awarded pilot grants to other teams to begin using Observer, with the goal of expanding the dataset into a national resource for improving health care. “These early projects are the start of a flywheel,” says Johnson. “As researchers generate new insights and recordings, the dataset will grow, letting us ask even more ambitious questions.”
With Observer linking video, audio, and transcripts to clinical data and electronic health records, researchers can now ask new questions: when laughter arises during a visit and whether it affects outcomes; how often clinicians look at patients versus their computer screens; how room layout or digital scribing technology changes communication; and how patients respond to explanations of diagnoses.
“This kind of multimodal evidence—combining video, audio, and medical records—creates opportunities across so many fields,” says Karen O’Connor, associate director of Johnson’s Artificial Intelligence for Ambulatory Care Innovation Lab. “By making this data available, we’re democratizing medical research and opening new paths to improving care.”
Materials in the Annenberg School for Communication Library Archives include thousands of TV scripts, the first issue of TV Guide, and interviews about the early days of HBO—which help to chronicle TV’s 100-year story.
Centering joy in AI development and implementation
PIK Professor Desmond Upton Patton—of Annenberg and SP2—and collaborators introduce a joy-informed framework designed to initiate conversations among engineers, designers, and researchers.
Winter Storm Fern brought icy and snowy conditions to the Northeast and other parts of the country over the weekend. Penn Today asks physicist Robert Carpick about the unique properties of ice, the science of curling, and how close we are to ‘nonslip’ ice.
Organizations like Penn’s Netter Center for Community Partnerships foster collaborations between Penn and public schools in the West Philadelphia community.
Penn receives national distinction for community engagement
The recognition by the American Council on Education and Carnegie Foundation for the Advancement of Teaching acknowledges Penn’s long-standing commitment to community-engaged scholarship and partnerships in West Philadelphia and beyond.
