Penn engineers have developed VLMgineer, an AI framework that can design, adapt, and deploy tools in the real world.
2 min. read
When Tianyu Li, a doctoral student in mechanical engineering and applied mechanics (MEAM) at the School of Engineering and Applied Science, and George Jiayuan Gao, former master’s student in robotics and current research engineer at Dyna, set out to study how robots might better interact with the physical world, they weren’t just thinking about smarter machines—they were thinking about tools.
“Humans don’t just rely on their bodies to solve problems,” Li explains. “We design tools to use.”
That simple idea sits at the core of a new research project from Li and Gao with their advisers, Nadia Figueroa, Shalini and Rajeev Misra Presidential Assistant Professor in MEAM, and Dinesh Jayaraman, assistant professor in computer and information science. The team’s project, VLMgineer, is a framework that uses artificial intelligence not just to “think,” but to design, adapt, and deploy tools in the real world.
Today’s large AI models are often associated with chatbots, systems that generate text, answer questions or write code. But Li’s research pushes beyond conversation into something more tangible: physical reasoning.
With VLMgineer, AI observes a task through visual input like a robot attempting to pick up scattered objects on a table. Instead of directly controlling the robot or rewriting its programming, the system takes a different approach: it suggests tool designs.
The AI proposes designs, simulates their effectiveness, iterates on improvements, and ultimately generates a refined solution. In some cases, those designs can be fabricated, using technologies like 3D printing, and tested in the real world. The system can then observe the outcome and continue learning.
It’s a cycle that mirrors how humans and animals learn: observe, experiment, adapt.
“This project is a reflection of my philosophy on how AI models should be used, not as all-knowing oracles, but as statistically plausible generators,” says Figueroa. “I see generative models as imperfect tools but they can be very powerful if combined with structured, verifiable approaches. This is exactly what we are doing with VLMgineer.”
The results have been both promising and surprising. In simulations, the AI generates multiple tool variations and optimizes them through evolutionary search. In real-world scenarios, VLMgineer may suggest using a scooper tool if given the task of gathering balls from a table. But sometimes, the ideas are unconventional or even physically unrealistic.
“That’s one of the challenges,” says Li. “The AI doesn’t always understand real-world constraints like materials or manufacturing limits yet.” Bridging that gap is a major focus of ongoing work.
When the Schuylkill swallowed the city: Lessons from Hurricane Ida’s historic flood
New Penn research shows that Hurricane Ida wasn’t a once-in-a-century anomaly but a preview of how climate change, urbanization, and aging infrastructure are rewriting flood risk.
The Fed explained: What it does and why it matters
Former Philadelphia Fed President Patrick Harker and financial historian Peter Conti-Brown, both Wharton professors, unpack the central bank’s origins, its unusual structure, and the quiet ways it shapes the economy
How population changes are impacting primary education worldwide
Research from Penn sociologist Emily Hannum and colleagues reveals regional trends in whether school-age populations are increasing, plateauing, or decreasing—and shows how different countries are responding.
A bioengineered bean gum from the lab of Penn Dental’s Henry Daniell is found to reduce the levels of three microbes associated with head and neck squamous cell cancer to almost zero, without affecting the beneficial bacteria normally found in the mouth.
Fighting oral cancer with bioengineered chewing gum
Research led by Penn Dental’s Henry Daniell shows that antiviral and antibacterial chewing gums reduce the levels of three microbes linked to worse outcomes in oral cancers, paving the way for more effective and affordable therapies.
