Artist-in-residence and visiting scholar Rebecca Kamen has blended AI and art to produce animated illustrations representing how a dyslexic brain interprets information.
A work that Penn artist-in-residence Rebecca Kamen produced for the show, “Dyslexic Dictionary” at Arion Press in San Francisco. Here, she reinterprets Ph.D. candidate Dale Zhou’s network visualization. (Image: Cat Fennell)
Communicating thoughts with words is considered a uniquely human evolutionary adaptation known as language processing. Fundamentally, it is an information exchange, a lot like data transfer between devices, but one riddled with discrete layers of complexity, as the ways in which our brains interpret and express ideas differ from person to person.
Learning challenges such as dyslexia are underpinned by these differences in language processing and can be characterized by difficulty learning and decoding information from written text.
Rebecca Kamen, artist-in-residence in the School of Arts & Sciences. (Image: Michele Corbman)
Artist-in-residence in Penn’s Department of Physics and Astronomy Rebecca Kamen has explored her personal relationship with dyslexia and information exchange to produce works that reflect elements of both her creative process and understanding of language. Kamen unveiled her latest exhibit at Arion Press Gallery in San Francisco, where nine artists with dyslexia were invited to produce imaginative interpretations of learning and experiencing language.
The artists were presented with several prompts in varying formats, including books, words, poems, quotes, articles, and even a single letter, and tasked with creating a dyslexic dictionary: an exploration of the ways in which their dyslexia empowered them to engage in information exchange in unique ways.
Undiagnosed dyslexia
“[For the exhibit], each artist selected a word representing the way they learn, and mine was ‘lens,’” explains Kamen. “It’s a word that captures how being dyslexic provides me with a unique perspective for viewing and interacting with the world.”
From an early age, Kamen enjoyed learning about the natural sciences and was excited about the process of discovery. She struggled, however, with reading at school, which initially presented an obstacle to achieving her dreams of becoming a teacher. “I had a difficult time getting into college,” says Kamen. “When I graduated high school, the word ‘dyslexia’ didn’t really exist, so I assumed everyone struggled with reading.”
RELATED
Kamen was diagnosed with dyslexia well into her tenure as a professor. “Most dyslexic people face challenges that may go unnoticed by others,” she says, “but they usually find creative ways to overcome them.”
This perspective on seeing and experiencing the world through the lens of dyslexia not only informed Kamen’s latest work for the exhibition “Dyslexic Dictionary,” but also showcased her background in merging art and science. For decades, Kamen’s work has investigated the intersection of the two, creating distinct ways of exploring new relationships and similarities.
“Artists and scientists are curious creatures always looking for patterns,” explains Kamen. “And that’s because patterns communicate larger insights about the world around us.”
The researchers studied different information-seeking approaches by monitoring how participants explore Wikipedia pages and categorically related these to two ideas rooted in philosophical understandings of learning: a “busybody,” who typically jumps between diverse ideas and collects loosely connected information; and a more purpose-driven “hunter,” who systematically ties in closely related concepts to fill their knowledge gaps.
(Left) A visual representation of the “hunter” and “busybody” information-seeking styles outlined in research from Dani Bassett, David Lydon-Staley and Perry Zurn. Circular nodes correspond to Wikipedia pages visited and lines represent the relationships between the pages. (Right) A visual representation of some of the topics corresponding to nodes; the line width shows the connection strength between associated ideas. (Image: David Lydon-Staley, Dani Bassett, Perry Zurn, Ann Sizemore, and Dale Zhou).
They used these classifications to inform their computational model, the knowledge network. This uses text and context to determine the degree of relatedness between the Wikipedia pages and their content—represented by dots connected with lines of varying thickness to illustrate the strength of association.
In an adaption of the knowledge network, Kamen was classified as a dancer, an archetype elaborated on in an accompanying review paper by Dale Zhou, a Ph.D. candidate in Bassett’s Complex Systems Lab, who had also collaborated with Kamen on “Reveal.”
Dale Zhou, a Ph.D. candidate in Bassett’s Complex Systems Lab. (Image: Eric Sucar)
“The dancer can be described as an individual that breaks away from the traditional pathways of investigation,” says Zhou. “Someone who takes leaps of creative imagination and in the process, produces new concepts and radically remodels knowledge networks.”
The artwork Kamen produced for “Dyslexic Dictionary” harkens back to “Reveal.” The knowledge network provided a way to depict Kamen’s distinct way of processing verbal information by analyzing transcripts of researcher interviews to construct a dataset of the keywords Kamen said.
From those interviews, Zhou mapped out the relatedness between each topic and clustered similar words together, graphically representing Kamen’s information-seeking and idea-generation process.
The network visualization depicting Kamen’s idea-generating process. Dots represent words that shaped Kamen’s artistic journey, lines indicate similarities between words, and colors represent communities of like ideas. (Image: Dale Zhou)
Zhou notes that, “the distance between the dots here shows how Rebecca’s able to bridge dissimilar ideas.” Kamen calls that, “one of the gifts of dyslexia that I’m learning to appreciate.”
Kamen took Zhou’s visualization and affixed several of her artworks to a few nodes, relating the keywords she had used in the interviews to recognizable images.
Kamen was one of nine artists with dyslexia who were invited to produce imaginative interpretations of learning and experiencing language for the exhibit, “Dyslexia Dictionary.” (Image: Cat Fennell)
AI and information processing
One of those artworks was Kamen’s Corona 3, a sculpture she produced during the pandemic that showcases her ability to connect disparate concepts. She relates models of the sun’s corona, the outermost layer of its atmosphere composed of hot plasma, to the spike proteins on the surface of SARS-CoV-2, which play a major role in its ability to infect host cells. Kamen highlights a similarity in how they interact with elements within their systems: The corona interacts with the sun’s powerful magnetic field and plasma, which cause solar winds that can affect bodies in the solar system and the spike proteins interact with the host cells.
Artists and scientists are curious creatures always looking for patterns. And that’s because patterns communicate larger insights about the world around us.
Artist-in-residence Rebecca Kamen
“People are fascinated when I show them these types of associations because we all have the ability to channel our curiosity and create new connections, but neurodiversity and dyslexia provide a catalyst for making these connections more apparent,” says Kamen. “What’s exciting here is that Dale was able to take the language I use to explain my process of discovery, and by connecting ideas, create something totally new.”
Zhou used artificial intelligence to transform Corona 3into an animation based on the interview transcripts and keywords from Kamen’s presentations.
An animation of Kamen’s sculpture, Corona 3. (Image: Dale Zhou)
Zhou created the animation with the open-source AI tool Stable Diffusion. Typically, AI models take in user-defined, labeled data points that train the model to do a specific assigned task. With Stable Diffusion, “the model has the whole internet to source images paired with text,” explains Zhou. “It’s trained by adding randomness to an incredibly large training dataset sequentially. The task is to learn how to reverse this addition by sequentially removing randomness, letting it reliably return new images similar to those in the training dataset.
“But there’s also a way you can make an image resemble text descriptions, which can change the image slowly over time and morph into something that sort of mimics the stream of consciousness that underpins Rebecca’s process.”
For their next piece, “Kamen’s Lens,” Zhou generated another animation, this time from a series of emails between Kamen and poet SJ Fowler that Fowler turned into poems.
Kamen used fragments of her email correspondence with poet SJ Fowler to inform her illustrations. Fowler then converted these emails into poems, which Zhou used as prompts for the AI-rendered final animation. (Image: Rebecca Kamen and Dale Zhou)
“By having artificial intelligence translate words into images, we appreciate a visual reimagining of language, which echoes some of the aspects of my experience with dyslexia in a profound new way,” says Kamen. “When I lecture and speak with students who struggle with reading the same way I did at their age, it’s nice to share that it can be seen as a gift and a way to create something new.”
Kamen hopes to use what she has learned from this process to inform her next exhibition at the American Philosophical Society, a special collection honoring unrecognized women in science. It will draw from existing knowledge bases, namely the discoveries and scientific breakthroughs these figures inspired, and reimagine them through Kamen’s lens incorporating art to highlight their significance for scientific and non-scientific audiences.
Homepage image: Kamen’s sculpture, Corona 3, 2020. Acrylic on mylar, 18 x 18 x 4 in., work that was sourced in part by colorized SARS-CoV-2 electron microscopy images captured by the NIAD/NIH team at Rocky Mountain Lab. (Image: Rebecca Kamen)
Deepfakes, digital doubles, and the law: Jennifer Rothman on protecting identity in the AI era
The evolution of AI technology may require more robust copyright laws and guardrails that protect people’s control of their own identities, says the Penn Carey Law professor.
Bold ideas and innovation on display at the Fall Research Expo
On Sept. 15, hundreds of posters were presented throughout Houston Hall at the annual Fall Research Expo, representing the research projects of 410 undergraduate students conducted through the Center for Undergraduate Research.
Exposure to air pollution worsens Alzheimer’s disease
New research from Penn Medicine finds living in areas with high concentration of air pollution is associated with increased buildup of amyloid and tau proteins in the brains of Alzheimer’s patients, accelerating cognitive decline.
