Columbia engineers make robots that learn by watching themselves

HQ Team

February 26, 2025: Robots can teach themselves how to move by watching their motions through a camera, similar to looking at a mirror, and also overcome damage to their bodies, Columbia Engineering researchers find.

“Like humans learning to dance by watching their mirror reflection, robots now use raw video to build kinematic self-awareness,” says study lead author Yuhang Hu, a doctoral student at the Creative Machines Lab at Columbia University.

“Our goal is a robot that understands its own body, adapts to damage, and learns new skills without constant human programming,” Hu said in a statement along with Hod Lipson, James and Sally Scapa, Professor of Innovation and chair of the Department of Mechanical Engineering, Columbia University.

A majority of the robots first learn to move in simulations. Once a robot moves in these virtual environments, it is released into the physical world where it can continue to learn. 

“The better and more realistic the simulator, the easier it is for the robot to make the leap from simulation into reality,” said Lipson. The researchers taught a robot how to create a simulator of itself simply by watching its motion through a camera. 

‘Wear and damage’

“This ability not only saves engineering effort, but also allows the simulation to continue and evolve with the robot as it undergoes wear, damage, and adaptation,” Lipson said.

The researchers developed a way for robots to autonomously model their 3D shapes using a single regular 2D camera. This breakthrough was driven by three brain-mimicking AI systems known as deep neural networks. 

These inferred 3D motion from 2D video, enabling the robot to understand and adapt to its movements. The new system could also identify alterations to the bodies of the robots, such as a bend in an arm, and help them adjust their motions to recover from this simulated damage.

Such adaptability might prove useful in a variety of real-world applications, the researchers said. 

More resilient robots

As we hand over more critical functions to robots, from manufacturing to medical care, we need these robots to be more resilient, the researchers said.

“We humans cannot afford to constantly baby these robots, repair broken parts and adjust performance. Robots need to learn to take care of themselves if they are going to become truly useful,” said Lipson. “That’s why self-modelling is so important.”

The researchers gave an example. 

“Imagine a robot vacuum or a personal assistant bot that notices its arm is bent after bumping into furniture,” Hu said. “Instead of breaking down or needing repair, it watches itself, adjusts how it moves and keeps working. This could make home robots more reliable — no constant reprogramming required.”

Another scenario involves a robot arm getting knocked out of alignment at a car factory. 

“Instead of halting production, it could watch itself, tweak its movements, and get back to welding—cutting downtime and costs,” Hu said. “This adaptability could make manufacturing  more resilient.”

Stick-figure simulations

About two decades ago, the same researcher’s robots were only able to create simple stick-figure-like simulations of themselves. By 2015, robots began creating higher-fidelity models using multiple cameras. 

In the current study, the robot was able to create a comprehensive kinematic model of itself using just a short video clip from a single regular camera. The researchers call this newfound ability “Kinematic Self-Awareness.”

“We humans are intuitively aware of our body; we can imagine ourselves in the future and visualize the consequences of our actions well before we perform those actions in reality,” said Lipson.

“Ultimately, we would like to imbue robots with a similar ability to imagine themselves because once you can imagine yourself in the future, there is no limit to what you can do.”